The problem was caused by a bad ordering between the restart of the ORTE level tcp connections (in the OOB - out-of-band communication) and the Open MPI level tcp connections (BTLs). Before this commit ORTE would shutdown and restart the OOB completely before the OMPI level restarted its tcp connections. What would happen is that a socket descriptor used by the OMPI level on checkpoint was assigned to the ORTE level on restart. But the OMPI level had no knowledge that the socket descriptor it was previously using has been recycled so it closed it on restart. This caused the ORTE level to break as the newly created socket descriptor was closed without its knowledge.
The fix is to have the OMPI level shutdown tcp connections, allow the ORTE level to restart, and then allow the OMPi level to restart its connections. This seems obvious, and I'm surprised that this bug has not cropped up sooner. I'm confident that this specific problem has been fixed with this commit.
Thanks to Eric Roman and Tamer El Sayed for their help in identifying this problem, and patience while I was fixing it.
* Add a new state {{{OPAL_CRS_RESTART_PRE}}}. This state identifies when we are on the down slope of the INC (finalize-like) which is useful when you want to close, but not reopen a component set for fear of interfering with a lower level.
* Use this new state in OMPI level coordination. Here we want to make sure to play well with both the OMPI/BTL/TCP and ORTE/OOB/TCP components.
* Update ft_event functions in PML and BML to handle the new restart state.
* Add an additional flag to the error output in OOB/TCP so we can see what the socket descriptor was on failure as this can be helpful in debugging.
This commit was SVN r18276.
{{{
svn merge -r 18218:18240 https://svn.open-mpi.org/svn/ompi/tmp/jjh-scratch .
}}}
Contains:
* Primarily a fix for a user reported problem where a cached file descriptor is causing a SIGPIPE on restart.
* Cleanup some small memory leaks from using mca_base_param_env_var() - Thanks Jeff
* Cleanup ORTE FT tool compilation in non-FT builds - Thanks Tim P.
* Cleanup mpi interface with missplaced {{{OPAL_CR_ENTER_LIBRARY}}} - Thanks Terry
* Some other sundry cleanup items all dealing with C/R functionality in the trunk.
This commit was SVN r18241.
Restore the "do-not-launch" functionality so users can test a mapping without launching it.
Add a "do-not-resolve" cmd line flag to mpirun so the opal/util/if.c code does not attempt to resolve network addresses, thus enabling a user to test a hostfile mapping without hanging on network resolve requests.
Add a function to hostfile to generate an ordered list of host names from a hostfile
This commit was SVN r18190.
1. applied prefix rule to functions and variables of RMAPS rank_file component
2. cleaned ompi_mpi_init.c from paffinity code
3. paffinity code moved to new opal/mca/paffinity/base/paffinity_base_service.c file
4. added opal_paffinity_slot_list mca parameter
This commit was SVN r18019.
mechanisms (such as epoll) if someone (ompi_mpi_init()) requests
otherwise. See big comment in opal/event/event.c for a full
explanation.
This commit was SVN r17956.
Greatly reduce the number of "foo" -> "opal_foo" symbol renames in the
libevent source, and instead greatly expand the event_rename.h file
that uses preprocessor macros to make all public symbols be
"opal_foo".
This commit was SVN r17923.
some cleanups and standardizations in the various */tools/*/
Makefile.am files. This commit:
* Somewhat simplify the tool Makefile.am's
* Makes the tool Makefile.am's consistent with each other (do similar
actions in similar ways)
* Update the tool Makefile.am's to remove old kruft that was required
by older versions of AM (trunk requires AM >=1.10)
This commit was SVN r17921.
* The opal_sys_timer_get_cycles() call was implemented for
Sparc v9 using inline assembly, but not in the assembly files.
This would only currently matter on Linux Sparc systems using
a compiler that didn't support inline assembly (not many of
those), but it should be there for completion.
* The linux timer component would always build on non-Alpha
platforms, rather than only building on platforms where
opal_sys_timer_get_cycles() was implemented. This would
only matter on a very narrow set of platforms that we don't
really support, but still, it could be more right. We now
only build the component on platforms where we have the
assembly call to get the cycle counter.
* Added a comment to opal/sys/timer.h to note that the linux
timer component needed to be updated if another platform was
added.
This should be harmless to commit. It will only really change
behaviors on platforms we don't have assembly support for, which
currently won't make it through configure. It really only matters
when (if?) we support atomic operations through libatomic_ops.
This commit was SVN r17887.
This commit lowers the priority of the darwin backtrace component
below that of the ''execinfo'' and ''stackprint'' components, which
will cause OS X Leopard to use the ''execinfo'' component. execinfo
utilizes a public API for printing the stacktrace. The ''darwin''
component uses some evil hacks and a not-so supported package from
Apple to print the stack trace.
This commit was SVN r17840.
environment variables in a consistent manner. These functions
retrieve the user and the temporary directories (based on the
system).
This commit was SVN r17815.
seemed to be the right thing(tm), but broke the Sun Studio C++
compiler under Linux (ticket 747).
This patch should allow inclusion into C and C++ from other header
files without problems.
This commit was SVN r17792.
The following SVN revision numbers were found above:
r17488 --> open-mpi/ompi@d53131f261
r17504 --> open-mpi/ompi@b22e8e7567
* Extension to the ESS framework to support C/R
* Fixed support for {{{snapc_base_establish_global_snapshot_dir}}}
* Fixed FileM support
* Misc. minor code modifications
There are some outstanding visability issues that I want to fix next.
This commit was SVN r17725.
