It turns that there is an incompatibility between the Cray PMI
library and the default configuration for building Open MPI (master).
To work around this, we now disable use of aprun for direct launch
of Open MPI jobs except under specific conditions.
The problem is that there are now (on master) packages getting
initialized that do not work properly across a fork operation.
As part of a constructor in the Cray PMI library, a fork operation
is done to simplify use of shared memory between the
processes in a job on the same node. This ends up thoroughly
messing up the Open MPI initialization process in the case
that dlopen support is enabled. The initialization process gets
about half-way through when the PMIX framework is opened and
components are loaded, which triggers the Cray PMI constructor
and hence the fork operation.
There are two workarounds for this:
1) configure Open MPI for Cray XE/XC systems using aprun with the
--disable-dlopen option
2) set the PMI_NO_FORK environment variable in the shell in which
the aprun command is run.
Without taking these measures, a Open MPI job will just hang at
job startup in the first attempt to "thread-shift" the PMIx
fence_nb operation. Additional hangs occur at shutdown if this
problem is worked around, again due to the insertion of a fork
operation halfway through the Open MPI initialization procedure.
This commit detects if the conditions that bring out the hang
situation are present, and if so, prints out a message and
aborts the job launch.
Note on systems using slurm, the PMI_NO_FORK environment variable
is set as part of the srun job launch, hence this issue is avoided
on those systems.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
An file might have been destroyed by an other task between
readdir() and stat(), so simply ignore stat() failure.
That typically occurs when one task is removing the job_session_dir
and an other task is still removing its proc_session_dir.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
opal_convertor_pack() might pack less bytes than requested,
so always set frag->segments[0].seg_len.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
PR open-mpi/ompi#2432 introduced a regression where configure
and build with --disable-dlopn caused build failure owing
to unresolved alps lli symbols in the libopal-pal shared library.
This commit fixes this problem.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Enhance the cray pmix component to set some OMPI internal
env. variables used to set some key/value pairs
on the MPI_INFO_ENV object. This allows more of the
ompi-tests ibm unit tests to pass when using aprun/srun
direct launch and Cray PMI.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
since pthreads are now mandatory, the MCA_BTL_TCP_SUPPORT_PROGRESS_THREAD
is always true and hence can be safely removed
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
Remove BTL_OPENIB_FAILOVER_ENABLED code in the openib btl source.
Remove the failover-specific files from the openib btl.
Update the openib/Makefile.am accordingly.
Remove the -enable-openib-failover config logic.
Signed-off-by: Steve Wise <swise@opengridcomputing.com>
This commit rewrites much of the btl/self component to fix a long
standing memory usage bug. Before this commit the prepare_src path
would always allocate a max send fragment (256kB). This caused the
rank to allocate 32 * 256k useless buffers from one send. This commit
makes the following changes:
- Add the MCA_BTL_FLAGS_GET flag by default. No reason not to set it.
- Reduce the eager limit, max send size, buffers per allocation, and
maximum buffer count per fragment size. These changes should have
no noticible affect on performance but should greatly reduce the
memory usage of the component.
- Implement the sendi function. This should reduce self send latency
somewhat.
- Rewrite prepare_src to never allocate a eager or max send fragment
for contiguous data.
- add_procs needs to return something in the peer array for the proc
self not just set the reachability bit. Now stores (void *) 1.
- Various cleanups. Removed and unused file.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The vader btl kept a per-peer registration cache to keep track of
attachments. This is not really a problem with small numbers of local
ranks but can be a problem with large SMP machines. To reduce the
footprint there is now one registration cache for all xpmem
attachments. This will probably increase the lookup time for large
transfers but is a worthwhile trade-off.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
the class system can be initialized/finalized as many times as we like,
so there is no more need to have opal_class_finalize() invoked in a destructor
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
libnl and libnl-3 are known to conflict with each other, so detect
and abort if these two libs are both used directly (e.g. Open MPI
uses libnl-3) or indirectly (e.g. libibverbs.so might depend on libnl)
Still not completely done as we need a better way of tracking the routed module being used down in the OOB - e.g., when a peer drops connection, we want to remove that route from all conduits that (a) use the OOB and (b) are routed, but we don't want to remove it from an OFI conduit.
