onto the backend daemons. By default, let mpirun only pack the app_context
info and send that to the backend daemons where the mapping will
be done. This significantly reduces the computational time on mpirun as it isn't
running up/down the topology tree computing thousands of binding
locations, and it reduces the launch message to a very small number of
bytes.
When running -novm, fall back to the old way of doing things
where mpirun computes the entire map and binding, and then sends
the full info to the backend daemon.
Add a new cmd line option/mca param --fwd-mpirun-port that allows
mpirun to dynamically select a port, but then passes that back to
all the other daemons so they will use that port as a static port
for their own wireup. In this mode, we no longer "phone home" directly
to mpirun, but instead use the static port to wireup at daemon
start. We then use the routing tree to rollup the initial
launch report, and limit the number of open sockets on mpirun's node.
Update ras simulator to track the new nidmap code
Cleanup some bugs in the nidmap regex code, and enhance the error message for not enough slots to include the host on which the problem is found.
Update gadget platform file
Initialize the range count when starting a new range
Fix the no-np case in managed allocation
Ensure DVM node usage gets cleaned up after each job
Update scaling.pl script to use --fwd-mpirun-port. Pre-connect the daemon to its parent during launch while we are otherwise waiting for the daemon's children to send their "phone home" rollup messages
Signed-off-by: Ralph Castain <rhc@open-mpi.org>
Check the exit status of major commands, as well as (optionally)
output the pwd and command being executed (when debugging). Also,
read the $debug variable from the environment; if it's set, go into
debugging mode (vs. requiring a modification to the script to enable
debugging mode).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
If $debug is set in the environment, use that. This allows enabling
debug mode without requiring an edit to the script.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
The filenames contain date/timestamps; if you compare those, the
tarball generated every night will *always* be new. Instead, separate
out the git hash from the old and new tarballs, and compare those.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
- Ensure that $to_delete is always defined
- Re-indent to 4 spaces for readability
- Don't only delete files -- it's ok to delete directories, too
- Print the directory from which we are deleting
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Just to make the scripts a little less error-prone. Also split up the
ssh/scp lines just for readability.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
- Don't use "-i" CLI option to perl; it's unnecessary here and causes
a warning
- Branch names may not be entirely letters (e.g., "v1.11"), so take
any character in the regexp to match the branch name
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Samples are taken after MPI_Init, and then again after MPI_Barrier. This allows the user to see memory consumption caused by add_procs, as well as any modex contribution from forming connections if pmix_base_async_modex is given.
Using the probe simply involves executing it via mpirun, with however many copies you want per node. Example:
$ mpirun -npernode 2 ./mpi_memprobe
Sampling memory usage after MPI_Init
Data for node rhc001
Daemon: 12.483398
Client: 6.514648
Data for node rhc002
Daemon: 11.865234
Client: 4.643555
Sampling memory usage after MPI_Barrier
Data for node rhc001
Daemon: 12.520508
Client: 6.576660
Data for node rhc002
Daemon: 11.879883
Client: 4.703125
Note that the client value on node rhc001 is larger - this is where rank=0 is housed, and apparently it gets a larger footprint for some reason.
Signed-off-by: Ralph Castain <rhc@open-mpi.org>
Revamp the event notification integration to rely on the PMIx event chaining and remove the duplicate chaining in OPAL. This ensures we get system-level events that target non-default handlers.
Restore the hostname entries for MPI-level error messages, but provide an MCA param (orte_hostname_cutoff) to remove them for large clusters where the memory footprint is problematic. Set the default at 1000 nodes in the job (not the allocation).
Begin first cut at memory profiler
Some minor cleanups of memprobe
Signed-off-by: Ralph Castain <rhc@open-mpi.org>
This is news to me: I didn't know that some distros do not set $HOME.
So use "~" instead, and only try to grep ~/.rpmmacros if it exists.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
* Point to local libfabric v1.4 install
* Add MPI C++ bindings
* Remove PSM support (if someone can install PSM/PSM2 libraries on the
build server, let's re-enable this)
Also change from -j8 to -j4 (the new AWS build instance only has 1
core / 2 hyperthreads).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Nightly snapshots will now be named:
openmpi-${BRANCHNAME}-${YYYYMMDDHHMM}-${SHORTHASH}.tar.${COMPRESSION}.
Fixes#2337
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Still not completely done as we need a better way of tracking the routed module being used down in the OOB - e.g., when a peer drops connection, we want to remove that route from all conduits that (a) use the OOB and (b) are routed, but we don't want to remove it from an OFI conduit.
