1
1
openmpi/orte/mca/pls
Ralph Castain b6196e8a39 When we can detect that a daemon has failed, then we would like to terminate the system without having it lock up. The "hang" is currently caused by the system attempting to send messages to the daemons (specifically, ordering them to kill their local procs and then terminate). Unfortunately, without some idea of which daemon has died, the system hangs while attempting to send a message to someone who is no longer alive.
This commit introduces the necessary logic to avoid that conflict. If a PLS component can identify that a daemon has failed, then we will set a flag indicating that fact. The xcast system will subsequently check that flag and, if it is set, will send all messages direct to the recipient. In the case of "kill local procs" and "terminate", the messages will go directly to each orted, thus bypassing any orted that has failed.

In addition, the xcast system will -not- wait for the messages to complete, but will return immediately (i.e., operate in non-blocking mode). Orterun will wait (via an event timer) for a period of time based on the number of daemons in the system to allow the messages to attempt to be delivered - at the end of that time, orterun will simply exit, alerting the user to the problem and -strongly- recommending they run orte-clean.

I could only test this on slurm for the case where all daemons unexpectedly died - srun apparently only executes its waitpid callback when all launched functions terminate. I have asked that Jeff integrate this capability into the OOB as he is working on it so that we execute it whenever a socket to an orted is unexpectedly closed. Meantime, the functionality will rarely get called, but at least the logic is available for anyone whose environment can support it.

This commit was SVN r16451.
2007-10-15 18:00:30 +00:00
..
base When we can detect that a daemon has failed, then we would like to terminate the system without having it lock up. The "hang" is currently caused by the system attempting to send messages to the daemons (specifically, ordering them to kill their local procs and then terminate). Unfortunately, without some idea of which daemon has died, the system hangs while attempting to send a message to someone who is no longer alive. 2007-10-15 18:00:30 +00:00
bproc These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC. 2007-10-05 19:48:23 +00:00
cnos Bring the CNOS component of the PLS back online 2007-06-12 22:17:05 +00:00
gridengine Modify the configure system so that gridengine support is only built in specific conditions: 2007-10-10 21:39:16 +00:00
lsf These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC. 2007-10-05 19:48:23 +00:00
poe reapply r15517 and r15520, which were removed in r15527 so that I could get 2007-07-20 02:34:29 +00:00
process These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC. 2007-10-05 19:48:23 +00:00
proxy - Export those components in correct way. 2007-08-08 16:20:17 +00:00
rsh These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC. 2007-10-05 19:48:23 +00:00
slurm When we can detect that a daemon has failed, then we would like to terminate the system without having it lock up. The "hang" is currently caused by the system attempting to send messages to the daemons (specifically, ordering them to kill their local procs and then terminate). Unfortunately, without some idea of which daemon has died, the system hangs while attempting to send a message to someone who is no longer alive. 2007-10-15 18:00:30 +00:00
submit These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC. 2007-10-05 19:48:23 +00:00
tm These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC. 2007-10-05 19:48:23 +00:00
xcpu reapply r15517 and r15520, which were removed in r15527 so that I could get 2007-07-20 02:34:29 +00:00
xgrid The previous hack to deal with Libtool not speaking Objective C stopped 2007-09-30 03:40:25 +00:00
Makefile.am Here is the major MAD-cure commit. I have written plenty about it, so I refer you here to those messages for a description of everything that was done. 2006-09-14 21:29:51 +00:00
pls_types.h Here is the major MAD-cure commit. I have written plenty about it, so I refer you here to those messages for a description of everything that was done. 2006-09-14 21:29:51 +00:00
pls.h Bring in the generalized xcast communication system along with the correspondingly revised orted launch. I will send a message out to developers explaining the basic changes. In brief: 2007-06-12 13:28:54 +00:00