Short version: there is a bug in OS X/Snow Leopard, but there is also
a bug in Open MPI. Fixing the bug in Open MPI is both trivial (a
1-line change) and avoids the bug in OS X. We'll file an OS X bug
report upstream with Apple, but it should no longer affect us here in
OMPI.
Fixes trac:2039.
More details:
Some background first:
1. IPv4 sockets can only accept incoming IPv4 connections. However,
IPv6 sockets can be configured to accept ''only'' incoming IPv6
connection, or ''both'' incoming IPv4 and IPv6 connections. An
IPv6 socket attribute sets which listening behavior is used.
1. IPv4 and IPv6 have different port namespaces. Hence, it is
permissable to bind a v4 socket to port X ''and'' also bind a v6
socket to that same port X on the same interface (assuming that
the v6 socket is only accepting incoming v6 connections).
Incoming v4 connections to port X on the interface should get
matched to the listening v4 socket; incoming v6 connections should
get matched to the listening v6 socket.
1. When v6 sockets accept ''both'' incoming v4 and v6 connections, it
should claim port X in both namespaces.
1. Linux's default behavior is to only allow one listening socket to
be bound to a given port (i.e., ''either'' a v6 or v4 socket to be
bound to a single port X -- not both). A v6 socket can listen for
both v4 and v6 incoming connections on that port, but still --
only one socket will be bound to that port.
1. Snow Leopard's default behavior is to share ports -- i.e., let
both a v4 and a v6 listening socket to be bound to port X
(assuming that the v6 socket is only accepting incoming v6
connections).
The TCP BTL creates a listening socket for each address family.
Hence, it creates a v4 listening socket on INADDR_ANY and a v6
listening socket on the v6 equivalent of INADDR_ANY. OMPI then
iteratively tries to find ports to listen on within the range of
[mca_btl_tcp_port_min, mca_btl_tcp_port_min + mca_btl_tcp_port_range).
On Linux, the v4 socket will be bound to port X and the v6 socket will
likely be bound to port Y (where X != Y). On Snow Leopard, the v4
socket will be bound to port X and the v6 socket may ''also'' be bound
to port X. Since the namespaces are separate, this shouldn't be a
problem.
However, Open MPI was accidentally setting the v6 listening behavior
to accept ''both'' v4 and v6 incoming connections. This is a trivial
thing to fix -- change a 0 to a 1 in the code. On Linux, this issue
didn't matter because the v4 and v6 sockets were on different ports.
So even though the v6 socket ''would'' have accepted incoming v4
connections, that never happened because OMPI would direct v4
connections to the v4 port.
But on Snow Leopard, the v4 and v6 listening ports could end up
sharing the same port number. As mentioned above, this ''shouldn't''
have been a problem, but it looks like Snow Leopard has the following
bugs:
* If a v4 socket is already bound to port X, we're pretty sure that a
v6 socket with the "accept both v4 and v6 incoming connections"
listening behavior should not be able to claim port X (because
there's already a v4 socket listening on X). However, Snow Leopard
would allow binding a v4 socket to port X, and then allow a v6
socket configured to allow incoming v4 and v6 connections to
''also'' be bound to port X.
* After binding the v6 socket to port X, Snow Leopard then lets
''another'' v4 socket ''also'' get bound to port X. Hence, there's
now '''three''' sockets all listening on port X.
This obviously led to mis-matched TCP connections, and things went
downhill from there.
That being said, Snow Leopard doesn't exhibit this behavior if v6
sockets only allow incoming v6 connections. And technically, that is
exactly the behavior we want (we want v6 sockets to only accept
incoming v6 connections). So if we just change the flag to make our
v6 listening socket us this behavior, the problem on OS X goes away.
That's what this commit does -- it changes a 0 to a 1, indicating
"only let this v6 socket allow incoming v6 connections."
That was simple, wasn't it?
This commit was SVN r22788.
The following Trac tickets were found above:
Ticket 2039 --> https://svn.open-mpi.org/trac/ompi/ticket/2039
1. The code that looks at btl_tcp_if_exclude before doing a
modex_send uses strcmp rather than strncmp. That means that
"lo0" gets sent even though "lo" is excluded.
2. The code that determines whether a particular local TCP
interface can connect to a particular remote interface doesn't
check for loopback interfaces. With this fix, users can now
enable "lo" and be assured that it will only be used for intra-
node communication.
This commit was SVN r22762.
In CMake 2.6 and earlier, this function add dependencies for targets and also link the target libraries automatically, but in CMake 2.8,this behavior has been changed, i.e. it will only add the dependencies but no link, which will cause linking errors at compilation time.
This commit was SVN r22405.
#if defined (c_plusplus)
defined (__cplusplus)
followed by
extern "C" {
and the closing counterpart by BEGIN_C_DECLS and END_C_DECLS.
Notable exceptions are:
- opal/include/opal_config_bottom.h:
This is our generated code, that itself defines BEGIN_C_DECL and
END_C_DECL
- ompi/mpi/cxx/mpicxx.h:
Here we do not include opal_config_bottom.h:
- Belongs to external code:
opal/mca/backtrace/darwin/MoreBacktrace/MoreDebugging/MoreBacktrace.c
opal/mca/backtrace/darwin/MoreBacktrace/MoreDebugging/MoreBacktrace.h
- opal/include/opal/prefetch.h:
Has C++ specific macros that are protected:
- Had #if ... } #endif _and_ END_C_DECLS (aka end up with 2x
END_C_DECLS)
ompi/mca/btl/openib/btl_openib.h
- opal/event/event.h has #ifdef __cplusplus as BEGIN_C_DECLS...
- opal/win32/ompi_process.h: had extern "C"\n {...
opal/win32/ompi_process.h: dito
- ompi/mca/btl/pcie/btl_pcie_lex.l: needed to add *_C_DECLS
ompi/mpi/f90/test/align_c.c: dito
- ompi/debuggers/msgq_interface.h: used #ifdef __cplusplus
- ompi/mpi/f90/xml/common-C.xsl: Amend
Tested on linux using --with-openib and --with-mx
The following do not contain either opal_config.h, orte_config.h or
ompi_config.h
(but possibly other header files, that include one of the above):
ompi/mca/bml/r2/bml_r2_ft.h
ompi/mca/btl/gm/btl_gm_endpoint.h
ompi/mca/btl/gm/btl_gm_proc.h
ompi/mca/btl/mx/btl_mx_endpoint.h
ompi/mca/btl/ofud/btl_ofud_endpoint.h
ompi/mca/btl/ofud/btl_ofud_frag.h
ompi/mca/btl/ofud/btl_ofud_proc.h
ompi/mca/btl/openib/btl_openib_mca.h
ompi/mca/btl/portals/btl_portals_endpoint.h
ompi/mca/btl/portals/btl_portals_frag.h
ompi/mca/btl/sctp/btl_sctp_endpoint.h
ompi/mca/btl/sctp/btl_sctp_proc.h
ompi/mca/btl/tcp/btl_tcp_endpoint.h
ompi/mca/btl/tcp/btl_tcp_ft.h
ompi/mca/btl/tcp/btl_tcp_proc.h
ompi/mca/btl/template/btl_template_endpoint.h
ompi/mca/btl/template/btl_template_proc.h
ompi/mca/btl/udapl/btl_udapl_eager_rdma.h
ompi/mca/btl/udapl/btl_udapl_endpoint.h
ompi/mca/btl/udapl/btl_udapl_mca.h
ompi/mca/btl/udapl/btl_udapl_proc.h
ompi/mca/mtl/mx/mtl_mx_endpoint.h
ompi/mca/mtl/mx/mtl_mx.h
ompi/mca/mtl/psm/mtl_psm_endpoint.h
ompi/mca/mtl/psm/mtl_psm.h
ompi/mca/pml/cm/pml_cm_component.h
ompi/mca/pml/csum/pml_csum_comm.h
ompi/mca/pml/dr/pml_dr_comm.h
ompi/mca/pml/dr/pml_dr_component.h
ompi/mca/pml/dr/pml_dr_endpoint.h
ompi/mca/pml/dr/pml_dr_recvfrag.h
ompi/mca/pml/example/pml_example.h
ompi/mca/pml/ob1/pml_ob1_comm.h
ompi/mca/pml/ob1/pml_ob1_component.h
ompi/mca/pml/ob1/pml_ob1_endpoint.h
ompi/mca/pml/ob1/pml_ob1_rdmafrag.h
ompi/mca/pml/ob1/pml_ob1_recvfrag.h
ompi/mca/pml/v/pml_v_output.h
opal/include/opal/prefetch.h
opal/mca/timer/aix/timer_aix.h
opal/util/qsort.h
test/support/components.h
This commit was SVN r21855.
The following SVN revision numbers were found above:
r2 --> open-mpi/ompi@58fdc18855
OMPI
and a language agnostic part in OPAL. The convertor is completely
moved into OPAL. This offers several benefits as described in RFC
http://www.open-mpi.org/community/lists/devel/2009/07/6387.php
namely:
- Fewer basic types (int* and float* types, boolean and wchar
- Fixing naming scheme to ompi-nomenclature.
- Usability outside of the ompi-layer.
- Due to the fixed nature of simple opal types, their information is
completely
known at compile time and therefore constified
- With fewer datatypes (22), the actual sizes of bit-field types may be
reduced
from 64 to 32 bits, allowing reorganizing the opal_datatype
structure, eliminating holes and keeping data required in convertor
(upon send/recv) in one cacheline...
This has implications to the convertor-datastructure and other parts
of the code.
- Several performance tests have been run, the netpipe latency does not
change with
this patch on Linux/x86-64 on the smoky cluster.
- Extensive tests have been done to verify correctness (no new
regressions) using:
1. mpi_test_suite on linux/x86-64 using clean ompi-trunk and
ompi-ddt:
a. running both trunk and ompi-ddt resulted in no differences
(except for MPI_SHORT_INT and MPI_TYPE_MIX_LB_UB do now run
correctly).
b. with --enable-memchecker and running under valgrind (one buglet
when run with static found in test-suite, commited)
2. ibm testsuite on linux/x86-64 using clean ompi-trunk and ompi-ddt:
all passed (except for the dynamic/ tests failed!! as trunk/MTT)
3. compilation and usage of HDF5 tests on Jaguar using PGI and
PathScale compilers.
4. compilation and usage on Scicortex.
- Please note, that for the heterogeneous case, (-m32 compiled
binaries/ompi), neither
ompi-trunk, nor ompi-ddt branch would successfully launch.
This commit was SVN r21641.
IPv4 and IPv6) is outside the legal boundaries. This fixes trac:1869.
This commit was SVN r21612.
The following Trac tickets were found above:
Ticket 1869 --> https://svn.open-mpi.org/trac/ompi/ticket/1869
for printing size_t use "%lu" and cast to (unsigned long).
This commit was SVN r21238.
The following SVN revision numbers were found above:
r21234 --> open-mpi/ompi@22b6177fb9
for us already.
* Slightly clarify the error message strings; now they match the new
error strings for btl_openib_ipaddr_in|exclude
This commit was SVN r21197.
subnet specifications (in addition to interface names). These
parameters now take a comma-delimited list of interfaces names and/or
a.b.c.d/x specifications (only IPv4 currently supported for subnet
specifications). For example:
mpirun --mca btl_tcp_if_include 10.10.30.0/8,eth0
This commit was SVN r21189.
OMPI_* to OPAL_*. This allows opal layer to be used more independent
from the whole of ompi.
NOTE: 9 "svn mv" operations immediately follow this commit.
This commit was SVN r21180.
- Delete unnecessary header files using
contrib/check_unnecessary_headers.sh after applying
patches, that include headers, being "lost" due to
inclusion in one of the now deleted headers...
In total 817 files are touched.
In ompi/mpi/c/ header files are moved up into the actual c-file,
where necessary (these are the only additional #include),
otherwise it is only deletions of #include (apart from the above
additions required due to notifier...)
- To get different MCAs (OpenIB, TM, ALPS), an earlier version was
successfully compiled (yesterday) on:
Linux locally using intel-11, gcc-4.3.2 and gcc-SVN + warnings enabled
Smoky cluster (x86-64 running Linux) using PGI-8.0.2 + warnings enabled
Lens cluster (x86-64 running Linux) using Pathscale-3.2 + warnings enabled
This commit was SVN r21096.
Anyway, this is blocking the move: do not include pml.h
if not really needed, aka none of the following used:
mca_pml
MCA_PML_CALL
OMPI_ANY_TAG
OMPI_ANY_SOURCE
OMPI_PROC_NULL
- Notable exceptions (deleting in one header->adding):
- ompi/mca/mtl/psm/
- ompi/mca/osc/rdma/
- ompi/mca/btl/openib/btl_openib_endpoint.c depended on
pml_base_sendreq.h
- Tested on Linux/x86-64, this time including make check
(thanks Jeff and Ralph)
This commit was SVN r20725.
get bitten by header depending on having already included
the corresponding [opal|orte|ompi]_config.h header.
When separating, things like [OPAL|ORTE|OMPI]_DECLSPEC
are missed.
Script to add the corresponding header in front of all following
(taking care of possible #ifdef HAVE_...)
- Including some minor cleanups to
- ompi/group/group.h -- include _after_ #ifndef OMPI_GROUP_H
- ompi/mca/btl/btl.h -- nclude _after_ #ifndef MCA_BTL_H
- ompi/mca/crcp/bkmrk/crcp_bkmrk_btl.c -- still no need for
orte/util/output.h
- ompi/mca/pml/dr/pml_dr_recvreq.c -- no need for mpool.h
- ompi/mca/btl/btl.h -- reorder to fit
- ompi/mca/bml/bml.h -- reorder to fit
- ompi/runtime/ompi_mpi_finalize.c -- reorder to fit
- ompi/request/request.h -- additionally need ompi/constants.h
- Tested on linux/x86-64
This commit was SVN r20720.
opal layer.
Add a check against a maximum (actually get rid of ifs internally to
opal_bitmap.c) -- the functionality to set the current maximum size
opal_bitmap_set_max_size() is currently only used in attribute.c
to set the maximum OMPI_FORTRAN_HANDLE_MAX...
Tested on linux/x86-64 with intel-tests with all_tests_no_perf_f
run with 6 procs.
Let's look into MTT as well...
This commit was SVN r20708.
anyhow -- if oob functionality is neededm then orte/mca/oob/oob.h
Nevertheless compiles fine with -Wimplicit-function-declaration
This commit was SVN r20641.
Only proc_info.h-internal include file is opal/dss/dss_types.h
- In one case (orte/util/hnp_contact.c) had to add proc_info.h again.
- Local compilation (Linux/x86_64) w/ -Wimplicit-function-declaration
works fine, no errors.
Again, let's have MTT the last word.
This commit was SVN r20631.
Often, orte/util/show_help.h is included, although no functionality
is required -- instead, most often opal_output.h, or
orte/mca/rml/rml_types.h
Please see orte_show_help_replacement.sh commited next.
- Local compilation (Linux/x86_64) w/ -Wimplicit-function-declaration
actually showed two *missing* #include "orte/util/show_help.h"
in orte/mca/odls/base/odls_base_default_fns.c and
in orte/tools/orte-top/orte-top.c
Manually added these.
Let's have MTT the last word.
This commit was SVN r20557.
parameters on the connecting side also. Also move define of IF_NAMESIZE
into if.h file. And lastly, add one verbose debug message which may be
useful if we run into other issues like this.
This commit fixes trac:1573.
This commit was SVN r19932.
The following Trac tickets were found above:
Ticket 1573 --> https://svn.open-mpi.org/trac/ompi/ticket/1573
