2007-07-23 22:36:33 +04:00
|
|
|
/*
|
|
|
|
|
* Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
|
|
|
|
* University Research and Technology
|
|
|
|
|
* Corporation. All rights reserved.
|
2009-03-31 20:23:27 +04:00
|
|
|
* Copyright (c) 2004-2009 The University of Tennessee and The University
|
2007-07-23 22:36:33 +04:00
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
|
* reserved.
|
|
|
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
|
|
|
* University of Stuttgart. All rights reserved.
|
|
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
|
* All rights reserved.
|
2009-01-11 05:30:00 +03:00
|
|
|
* Copyright (c) 2007 Cisco Systems, Inc. All rights reserved.
|
2007-07-23 22:36:33 +04:00
|
|
|
* Copyright (c) 2007 Los Alamos National Security, LLC. All rights
|
|
|
|
|
* reserved.
|
2009-01-30 21:50:10 +03:00
|
|
|
* Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved.
|
2007-07-23 22:36:33 +04:00
|
|
|
* $COPYRIGHT$
|
|
|
|
|
*
|
|
|
|
|
* Additional copyrights may follow
|
|
|
|
|
*
|
|
|
|
|
* $HEADER$
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include "orte_config.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/constants.h"
|
2007-07-23 22:36:33 +04:00
|
|
|
|
|
|
|
|
#include <stdio.h>
|
2009-03-31 20:23:27 +04:00
|
|
|
#include <stddef.h>
|
2007-07-23 22:36:33 +04:00
|
|
|
#include <ctype.h>
|
|
|
|
|
#ifdef HAVE_UNISTD_H
|
|
|
|
|
#include <unistd.h>
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef HAVE_NETDB_H
|
|
|
|
|
#include <netdb.h>
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef HAVE_SYS_PARAM_H
|
|
|
|
|
#include <sys/param.h>
|
|
|
|
|
#endif
|
|
|
|
|
#include <fcntl.h>
|
|
|
|
|
#include <errno.h>
|
|
|
|
|
#include <signal.h>
|
2009-01-12 22:12:58 +03:00
|
|
|
#ifdef HAVE_TIME_H
|
|
|
|
|
#include <time.h>
|
|
|
|
|
#endif
|
2007-07-23 22:36:33 +04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
#include "opal/event/event.h"
|
|
|
|
|
#include "opal/mca/base/base.h"
|
2009-02-14 05:26:12 +03:00
|
|
|
#include "opal/util/output.h"
|
2007-07-23 22:36:33 +04:00
|
|
|
#include "opal/util/opal_environ.h"
|
|
|
|
|
#include "opal/runtime/opal.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "opal/runtime/opal_progress.h"
|
2007-07-23 22:36:33 +04:00
|
|
|
#include "opal/mca/base/mca_base_param.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "opal/dss/dss.h"
|
2008-12-22 23:23:05 +03:00
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
#include "orte/util/proc_info.h"
|
|
|
|
|
#include "orte/util/session_dir.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/util/name_fns.h"
|
2007-07-23 22:36:33 +04:00
|
|
|
|
|
|
|
|
#include "orte/mca/errmgr/errmgr.h"
|
|
|
|
|
#include "orte/mca/rml/rml.h"
|
2009-02-14 05:26:12 +03:00
|
|
|
#include "orte/mca/rml/rml_types.h"
|
2007-07-23 22:36:33 +04:00
|
|
|
#include "orte/mca/odls/odls.h"
|
2008-12-10 23:40:47 +03:00
|
|
|
#include "orte/mca/odls/base/base.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/plm/plm.h"
|
2008-06-03 18:23:04 +04:00
|
|
|
#include "orte/mca/plm/base/plm_private.h"
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
#include "orte/mca/routed/routed.h"
|
2008-12-22 23:23:05 +03:00
|
|
|
#include "orte/mca/ess/ess.h"
|
2007-07-23 22:36:33 +04:00
|
|
|
|
2008-12-10 22:18:36 +03:00
|
|
|
#include "orte/mca/odls/base/odls_private.h"
|
|
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
#include "orte/runtime/runtime.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/runtime/orte_globals.h"
|
|
|
|
|
#include "orte/runtime/orte_wait.h"
|
2007-07-23 22:36:33 +04:00
|
|
|
|
|
|
|
|
#include "orte/orted/orted.h"
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Globals
|
|
|
|
|
*/
|
|
|
|
|
static int process_commands(orte_process_name_t* sender,
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_buffer_t *buffer,
|
2007-07-23 22:36:33 +04:00
|
|
|
orte_rml_tag_t tag);
|
|
|
|
|
|
2009-01-12 22:12:58 +03:00
|
|
|
/* instantiate this - it is shared via orted.h */
|
|
|
|
|
struct timeval orte_daemon_msg_recvd;
|
|
|
|
|
|
|
|
|
|
static struct timeval mesg_recvd={0,0};
|
2008-04-14 22:26:08 +04:00
|
|
|
|
2009-01-27 22:13:56 +03:00
|
|
|
static void send_relay(opal_buffer_t *buf)
|
2008-02-28 04:57:57 +03:00
|
|
|
{
|
2008-05-06 02:32:25 +04:00
|
|
|
opal_list_t recips;
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_list_item_t *item;
|
2009-01-27 22:13:56 +03:00
|
|
|
orte_routed_tree_t *nm;
|
|
|
|
|
orte_process_name_t target;
|
2008-02-28 04:57:57 +03:00
|
|
|
int ret;
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-02-29 18:18:45 +03:00
|
|
|
"%s orte:daemon:send_relay",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2008-02-29 18:18:45 +03:00
|
|
|
|
2008-05-06 02:32:25 +04:00
|
|
|
/* get the list of next recipients from the routed module */
|
|
|
|
|
OBJ_CONSTRUCT(&recips, opal_list_t);
|
|
|
|
|
/* ignore returned parent vpid - we don't care here */
|
2009-01-27 22:13:56 +03:00
|
|
|
orte_routed.get_routing_tree(&recips);
|
2008-02-29 18:18:45 +03:00
|
|
|
|
2008-05-06 02:32:25 +04:00
|
|
|
/* if list is empty, nothing for us to do */
|
|
|
|
|
if (opal_list_is_empty(&recips)) {
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-05-06 02:32:25 +04:00
|
|
|
"%s orte:daemon:send_relay - recipient list is empty!",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2008-05-06 02:32:25 +04:00
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
2009-01-27 22:13:56 +03:00
|
|
|
/* send the message to each recipient on list, deconstructing it as we go */
|
|
|
|
|
target.jobid = ORTE_PROC_MY_NAME->jobid;
|
|
|
|
|
while (NULL != (item = opal_list_remove_first(&recips))) {
|
|
|
|
|
nm = (orte_routed_tree_t*)item;
|
|
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2009-01-27 22:13:56 +03:00
|
|
|
"%s orte:daemon:send_relay sending relay msg to %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_VPID_PRINT(nm->vpid)));
|
2008-02-29 18:18:45 +03:00
|
|
|
|
2009-01-27 22:13:56 +03:00
|
|
|
target.vpid = nm->vpid;
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > (ret = orte_rml.send_buffer(&target, buf, ORTE_RML_TAG_DAEMON, 0))) {
|
2009-01-27 22:13:56 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
2008-05-06 02:32:25 +04:00
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
CLEANUP:
|
2008-02-28 04:57:57 +03:00
|
|
|
/* cleanup */
|
2008-05-06 02:32:25 +04:00
|
|
|
OBJ_DESTRUCT(&recips);
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void orte_daemon_recv(int status, orte_process_name_t* sender,
|
|
|
|
|
opal_buffer_t *buffer, orte_rml_tag_t tag,
|
|
|
|
|
void* cbdata)
|
|
|
|
|
{
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-02-28 04:57:57 +03:00
|
|
|
"%s orted_recv_cmd: received message from %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_NAME_PRINT(sender)));
|
2007-07-23 22:36:33 +04:00
|
|
|
|
2009-01-12 22:12:58 +03:00
|
|
|
if (orte_timing) {
|
|
|
|
|
/* pickup the time the message was recvd by this daemon */
|
|
|
|
|
gettimeofday(&mesg_recvd, NULL);
|
|
|
|
|
}
|
|
|
|
|
|
2008-02-28 22:58:32 +03:00
|
|
|
/* don't process this right away - we need to get out of the recv before
|
|
|
|
|
* we process the message as it may ask us to do something that involves
|
|
|
|
|
* more messaging! Instead, setup an event so that the message gets processed
|
|
|
|
|
* as soon as we leave the recv.
|
|
|
|
|
*
|
|
|
|
|
* The macro makes a copy of the buffer, which we release when processed - the incoming
|
|
|
|
|
* buffer, however, is NOT released here, although its payload IS transferred
|
|
|
|
|
* to the message buffer for later processing
|
|
|
|
|
*/
|
|
|
|
|
ORTE_MESSAGE_EVENT(sender, buffer, tag, orte_daemon_cmd_processor);
|
2008-02-28 04:57:57 +03:00
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-02-28 04:57:57 +03:00
|
|
|
"%s orted_recv_cmd: reissued recv",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.
Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...
Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.
After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.
Thanks you Jeff! :-/
This commit was SVN r18611.
2008-06-06 23:36:27 +04:00
|
|
|
|
|
|
|
|
static int num_recursions=0;
|
|
|
|
|
static int wait_time=1;
|
|
|
|
|
#define MAX_RECURSIONS 24
|
|
|
|
|
|
2008-02-28 22:58:32 +03:00
|
|
|
void orte_daemon_cmd_processor(int fd, short event, void *data)
|
2007-07-23 22:36:33 +04:00
|
|
|
{
|
2008-02-28 22:58:32 +03:00
|
|
|
orte_message_event_t *mev = (orte_message_event_t*)data;
|
2008-12-10 22:18:36 +03:00
|
|
|
orte_process_name_t *sender = &(mev->sender);
|
2008-02-28 22:58:32 +03:00
|
|
|
opal_buffer_t *buffer = mev->buffer;
|
2009-04-30 19:08:02 +04:00
|
|
|
opal_buffer_t relay_buf;
|
2008-05-06 02:32:25 +04:00
|
|
|
orte_rml_tag_t tag = mev->tag, target_tag;
|
|
|
|
|
orte_jobid_t job;
|
2007-07-23 22:36:33 +04:00
|
|
|
int ret;
|
2009-03-31 20:23:27 +04:00
|
|
|
ptrdiff_t unpack_rel, save_rel;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_std_cntr_t n;
|
|
|
|
|
orte_daemon_cmd_flag_t command;
|
|
|
|
|
|
Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.
Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...
Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.
After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.
Thanks you Jeff! :-/
This commit was SVN r18611.
2008-06-06 23:36:27 +04:00
|
|
|
/* check to see if we are in a progress recursion */
|
2009-05-04 15:07:40 +04:00
|
|
|
if (ORTE_PROC_IS_DAEMON && 1 < (ret = opal_progress_recursion_depth())) {
|
Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.
Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...
Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.
After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.
Thanks you Jeff! :-/
This commit was SVN r18611.
2008-06-06 23:36:27 +04:00
|
|
|
/* if we are in a recursion, we want to repost the message event
|
|
|
|
|
* so the progress engine can work its way back up to the top
|
|
|
|
|
* of the stack. Given that this could happen multiple times,
|
|
|
|
|
* we have to be careful to increase the time we wait so that
|
|
|
|
|
* we provide enough time - but not more time than necessary - for
|
|
|
|
|
* the stack to clear
|
|
|
|
|
*/
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.
Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...
Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.
After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.
Thanks you Jeff! :-/
This commit was SVN r18611.
2008-06-06 23:36:27 +04:00
|
|
|
"%s orte:daemon:cmd:processor in recursion depth %d\n\treposting %s for tag %ld",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.
Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...
Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.
After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.
Thanks you Jeff! :-/
This commit was SVN r18611.
2008-06-06 23:36:27 +04:00
|
|
|
ret,
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(sender),
|
Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.
Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...
Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.
After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.
Thanks you Jeff! :-/
This commit was SVN r18611.
2008-06-06 23:36:27 +04:00
|
|
|
(long)(tag)));
|
|
|
|
|
if (MAX_RECURSIONS < num_recursions) {
|
|
|
|
|
/* we need to abort if we get too far down this path */
|
|
|
|
|
opal_output(0, "%s ORTED_CMD_PROCESSOR: STUCK IN INFINITE LOOP - ABORTING",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.
Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...
Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.
After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.
Thanks you Jeff! :-/
This commit was SVN r18611.
2008-06-06 23:36:27 +04:00
|
|
|
OBJ_RELEASE(mev);
|
|
|
|
|
/* make sure our local procs are dead - but don't update their state
|
|
|
|
|
* on the HNP as this may be redundant
|
|
|
|
|
*/
|
|
|
|
|
orte_odls.kill_local_procs(ORTE_JOBID_WILDCARD, false);
|
|
|
|
|
|
|
|
|
|
/* do -not- call finalize as this will send a message to the HNP
|
|
|
|
|
* indicating clean termination! Instead, just forcibly cleanup
|
|
|
|
|
* the local session_dir tree and abort
|
|
|
|
|
*/
|
|
|
|
|
orte_session_dir_cleanup(ORTE_JOBID_WILDCARD);
|
|
|
|
|
|
|
|
|
|
abort();
|
|
|
|
|
}
|
|
|
|
|
wait_time = wait_time * 2;
|
|
|
|
|
++num_recursions;
|
|
|
|
|
ORTE_MESSAGE_EVENT_DELAY(wait_time, mev);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
wait_time = 1;
|
|
|
|
|
num_recursions = 0;
|
|
|
|
|
|
2009-05-04 15:07:40 +04:00
|
|
|
if (orte_timing && ORTE_PROC_IS_HNP) {
|
2009-01-12 22:12:58 +03:00
|
|
|
/* if we are doing timing, and we are the HNP, then the message doesn't come
|
|
|
|
|
* through the RML recv, so we have to pickup the recv time here
|
|
|
|
|
*/
|
|
|
|
|
gettimeofday(&mesg_recvd, NULL);
|
|
|
|
|
}
|
|
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-02-28 04:57:57 +03:00
|
|
|
"%s orte:daemon:cmd:processor called by %s for tag %ld",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_NAME_PRINT(sender),
|
2008-02-28 04:57:57 +03:00
|
|
|
(long)(tag)));
|
|
|
|
|
|
|
|
|
|
/* save the original buffer pointers */
|
2009-03-31 20:23:27 +04:00
|
|
|
unpack_rel = buffer->unpack_ptr - buffer->base_ptr;
|
2007-07-23 22:36:33 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* unpack the initial command */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &command, &n, ORTE_DAEMON_CMD))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
2009-05-07 00:11:28 +04:00
|
|
|
#if OPAL_ENABLE_DEBUG
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s got message buffer from file %s line %d\n",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), mev->file, mev->line);
|
2008-03-03 19:06:47 +03:00
|
|
|
#endif
|
2008-02-28 04:57:57 +03:00
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2007-07-23 22:36:33 +04:00
|
|
|
|
2008-05-06 02:32:25 +04:00
|
|
|
/* see if this is a "process-and-relay" command - i.e., an xcast is underway */
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_DAEMON_PROCESS_AND_RELAY_CMD == command) {
|
2008-05-06 02:32:25 +04:00
|
|
|
/* get the target jobid and tag */
|
2008-02-28 04:57:57 +03:00
|
|
|
n = 1;
|
2008-05-06 02:32:25 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &job, &n, ORTE_JOBID))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2008-05-06 02:32:25 +04:00
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &target_tag, &n, ORTE_RML_TAG))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2008-05-06 02:32:25 +04:00
|
|
|
/* save this buffer location */
|
2009-03-31 20:23:27 +04:00
|
|
|
save_rel = buffer->unpack_ptr - buffer->base_ptr;
|
2008-12-10 22:18:36 +03:00
|
|
|
/* unpack the command that will actually be executed */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &command, &n, ORTE_DAEMON_CMD))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* is this an add-procs cmd? */
|
|
|
|
|
if (ORTE_DAEMON_ADD_LOCAL_PROCS == command) {
|
2009-04-30 19:08:02 +04:00
|
|
|
/* store the time the cmd was recvd */
|
|
|
|
|
if (orte_timing) {
|
|
|
|
|
orte_daemon_msg_recvd.tv_sec = mesg_recvd.tv_sec;
|
|
|
|
|
orte_daemon_msg_recvd.tv_usec = mesg_recvd.tv_usec;
|
|
|
|
|
}
|
|
|
|
|
/* cmd contains daemon update info - process it */
|
2008-12-10 22:18:36 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls_base_default_update_daemon_info(buffer))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* flag this location */
|
2009-03-31 20:23:27 +04:00
|
|
|
save_rel = buffer->unpack_ptr - buffer->base_ptr;
|
2008-12-10 22:18:36 +03:00
|
|
|
}
|
|
|
|
|
|
2009-04-30 19:08:02 +04:00
|
|
|
/* setup the relay buffer */
|
|
|
|
|
OBJ_CONSTRUCT(&relay_buf, opal_buffer_t);
|
|
|
|
|
/* rewind the buffer to the beginning */
|
|
|
|
|
buffer->unpack_ptr = buffer->base_ptr + unpack_rel;
|
|
|
|
|
/* copy everything to the relay buffer */
|
|
|
|
|
opal_dss.copy_payload(&relay_buf, buffer);
|
|
|
|
|
/* do the relay */
|
|
|
|
|
send_relay(&relay_buf);
|
|
|
|
|
/* cleanup */
|
|
|
|
|
OBJ_DESTRUCT(&relay_buf);
|
|
|
|
|
|
2009-03-01 04:24:02 +03:00
|
|
|
/* rewind the buffer to the right place for processing the cmd */
|
2009-03-31 20:23:27 +04:00
|
|
|
buffer->unpack_ptr = buffer->base_ptr + save_rel;
|
2009-03-01 04:24:02 +03:00
|
|
|
|
|
|
|
|
/* process the command */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = process_commands(sender, buffer, tag))) {
|
|
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
|
|
|
|
"%s orte:daemon:cmd:processor failed on error %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), ORTE_ERROR_NAME(ret)));
|
2009-03-01 04:24:02 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* done */
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
} else {
|
|
|
|
|
/* rewind the buffer so we can process it correctly */
|
2009-03-31 20:23:27 +04:00
|
|
|
buffer->unpack_ptr = buffer->base_ptr + unpack_rel;
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* process the command */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = process_commands(sender, buffer, tag))) {
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-02-28 04:57:57 +03:00
|
|
|
"%s orte:daemon:cmd:processor failed on error %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), ORTE_ERROR_NAME(ret)));
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-02-28 04:57:57 +03:00
|
|
|
"%s orte:daemon:cmd:processor: processing commands completed",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2008-02-29 18:18:45 +03:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
CLEANUP:
|
2008-02-28 22:58:32 +03:00
|
|
|
OBJ_RELEASE(mev);
|
2008-04-17 00:41:00 +04:00
|
|
|
/* reissue the non-blocking receive */
|
|
|
|
|
ret = orte_rml.recv_buffer_nb(ORTE_NAME_WILDCARD, ORTE_RML_TAG_DAEMON,
|
|
|
|
|
ORTE_RML_NON_PERSISTENT, orte_daemon_recv, NULL);
|
|
|
|
|
if (ret != ORTE_SUCCESS && ret != ORTE_ERR_NOT_IMPLEMENTED) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
}
|
|
|
|
|
|
2008-02-28 22:58:32 +03:00
|
|
|
return;
|
2009-03-31 20:23:27 +04:00
|
|
|
}
|
2007-07-23 22:36:33 +04:00
|
|
|
|
|
|
|
|
static int process_commands(orte_process_name_t* sender,
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_buffer_t *buffer,
|
2007-07-23 22:36:33 +04:00
|
|
|
orte_rml_tag_t tag)
|
|
|
|
|
{
|
|
|
|
|
orte_daemon_cmd_flag_t command;
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_buffer_t *relay_msg;
|
2007-07-23 22:36:33 +04:00
|
|
|
int ret;
|
|
|
|
|
orte_std_cntr_t n;
|
|
|
|
|
int32_t signal;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_jobid_t job;
|
2007-07-23 22:36:33 +04:00
|
|
|
orte_rml_tag_t target_tag;
|
2008-05-01 23:19:34 +04:00
|
|
|
char *contact_info;
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_buffer_t *answer;
|
2007-07-23 22:36:33 +04:00
|
|
|
orte_rml_cmd_flag_t rml_cmd;
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
orte_job_t *jdata;
|
2008-12-22 23:23:05 +03:00
|
|
|
orte_process_name_t proc, proc2;
|
2008-12-10 23:40:47 +03:00
|
|
|
int32_t status;
|
2008-12-22 23:23:05 +03:00
|
|
|
orte_process_name_t *return_addr;
|
|
|
|
|
int32_t num_replies;
|
|
|
|
|
bool hnp_accounted_for;
|
|
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
/* unpack the command */
|
|
|
|
|
n = 1;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &command, &n, ORTE_DAEMON_CMD))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* now process the command locally */
|
|
|
|
|
switch(command) {
|
|
|
|
|
|
2008-06-19 17:48:26 +04:00
|
|
|
/**** NULL ****/
|
|
|
|
|
case ORTE_DAEMON_NULL_CMD:
|
|
|
|
|
ret = ORTE_SUCCESS;
|
|
|
|
|
break;
|
|
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
/**** KILL_LOCAL_PROCS ****/
|
|
|
|
|
case ORTE_DAEMON_KILL_LOCAL_PROCS:
|
2008-02-28 04:57:57 +03:00
|
|
|
/* unpack the jobid */
|
2007-07-23 22:36:33 +04:00
|
|
|
n = 1;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &job, &n, ORTE_JOBID))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls.kill_local_procs(job, true))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
/**** SIGNAL_LOCAL_PROCS ****/
|
|
|
|
|
case ORTE_DAEMON_SIGNAL_LOCAL_PROCS:
|
2008-02-28 04:57:57 +03:00
|
|
|
/* unpack the jobid */
|
2007-07-23 22:36:33 +04:00
|
|
|
n = 1;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &job, &n, ORTE_JOBID))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* get the signal */
|
|
|
|
|
n = 1;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &signal, &n, OPAL_INT32))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2009-01-30 21:50:10 +03:00
|
|
|
|
|
|
|
|
/* Convert SIGTSTP to SIGSTOP so we can suspend a.out */
|
|
|
|
|
if (SIGTSTP == signal) {
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
|
|
|
|
opal_output(0, "%s orted_cmd: converted SIGTSTP to SIGSTOP before delivering",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2009-01-30 21:50:10 +03:00
|
|
|
}
|
|
|
|
|
signal = SIGSTOP;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
|
|
|
|
opal_output(0, "%s orted_cmd: received signal_local_procs, delivering signal %d",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
2009-01-30 21:50:10 +03:00
|
|
|
signal);
|
|
|
|
|
}
|
|
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
/* signal them */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls.signal_local_procs(NULL, signal))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
/**** ADD_LOCAL_PROCS ****/
|
|
|
|
|
case ORTE_DAEMON_ADD_LOCAL_PROCS:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received add_local_procs",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
/* launch the processes */
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls.launch_local_procs(buffer))) {
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-02-28 04:57:57 +03:00
|
|
|
"%s orted:comm:add_procs failed to launch on error %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), ORTE_ERROR_NAME(ret)));
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
2008-05-01 23:19:34 +04:00
|
|
|
/**** TREE_SPAWN ****/
|
|
|
|
|
case ORTE_DAEMON_TREE_SPAWN:
|
2008-04-14 22:26:08 +04:00
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received tree_spawn",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-04-14 22:26:08 +04:00
|
|
|
}
|
2008-04-30 23:49:53 +04:00
|
|
|
/* if the PLM supports remote spawn, pass it all along */
|
|
|
|
|
if (NULL != orte_plm.remote_spawn) {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_plm.remote_spawn(buffer))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
}
|
|
|
|
|
} else {
|
2009-03-06 00:50:47 +03:00
|
|
|
opal_output(0, "%s remote spawn is NULL!", ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-04-30 23:49:53 +04:00
|
|
|
}
|
2008-04-14 22:26:08 +04:00
|
|
|
break;
|
|
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
/**** DELIVER A MESSAGE TO THE LOCAL PROCS ****/
|
|
|
|
|
case ORTE_DAEMON_MESSAGE_LOCAL_PROCS:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received message_local_procs",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* unpack the jobid of the procs that are to receive the message */
|
|
|
|
|
n = 1;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &job, &n, ORTE_JOBID))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* unpack the tag where we are to deliver the message */
|
|
|
|
|
n = 1;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &target_tag, &n, ORTE_RML_TAG))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_debug_output,
|
2008-04-17 00:41:00 +04:00
|
|
|
"%s orted:comm:message_local_procs delivering message to job %s tag %d",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_JOBID_PRINT(job), (int)target_tag));
|
2008-04-17 00:41:00 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
relay_msg = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
opal_dss.copy_payload(relay_msg, buffer);
|
2007-07-23 22:36:33 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* if job=my_jobid, then this message is for us and not for our children */
|
|
|
|
|
if (ORTE_PROC_MY_NAME->jobid == job) {
|
2007-07-23 22:36:33 +04:00
|
|
|
/* if the target tag is our xcast_barrier or rml_update, then we have
|
|
|
|
|
* to handle the message as a special case. The RML has logic in it
|
|
|
|
|
* intended to make it easier to use. This special logic mandates that
|
|
|
|
|
* any message we "send" actually only goes into the queue for later
|
|
|
|
|
* transmission. Thus, since we are already in a recv when we enter
|
|
|
|
|
* the "process_commands" function, any attempt to "send" the relay
|
|
|
|
|
* buffer to ourselves will only be added to the queue - it won't
|
|
|
|
|
* actually be delivered until *after* we conclude the processing
|
|
|
|
|
* of the current recv.
|
|
|
|
|
*
|
|
|
|
|
* The problem here is that, for messages where we need to relay
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
* them along the orted chain, the rml_update
|
|
|
|
|
* message contains contact info we may well need in order to do
|
2007-07-23 22:36:33 +04:00
|
|
|
* the relay! So we need to process those messages immediately.
|
|
|
|
|
* The only way to accomplish that is to (a) detect that the
|
|
|
|
|
* buffer is intended for those tags, and then (b) process
|
|
|
|
|
* those buffers here.
|
|
|
|
|
*
|
|
|
|
|
*/
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
if (ORTE_RML_TAG_RML_INFO_UPDATE == target_tag) {
|
2007-07-23 22:36:33 +04:00
|
|
|
n = 1;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(relay_msg, &rml_cmd, &n, ORTE_RML_CMD))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
/* initialize the routes to my peers - this will update the number
|
2009-03-06 00:56:03 +03:00
|
|
|
* of daemons in the system (i.e., orte_process_info.num_procs) as
|
2008-02-28 04:57:57 +03:00
|
|
|
* this might have changed
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_routed.init_routes(ORTE_PROC_MY_NAME->jobid, relay_msg))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
/* just deliver it to ourselves */
|
2008-02-28 04:57:57 +03:00
|
|
|
if ((ret = orte_rml.send_buffer(ORTE_PROC_MY_NAME, relay_msg, target_tag, 0)) < 0) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
} else {
|
|
|
|
|
ret = ORTE_SUCCESS;
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_progress(); /* give us a chance to move the message along */
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
/* must be for our children - deliver the message */
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls.deliver_message(job, relay_msg, target_tag))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
OBJ_RELEASE(relay_msg);
|
2007-07-23 22:36:33 +04:00
|
|
|
break;
|
|
|
|
|
|
2008-12-10 23:40:47 +03:00
|
|
|
/**** WAITPID_FIRED COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_WAITPID_FIRED:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
|
|
|
|
opal_output(0, "%s orted_cmd: received waitpid_fired cmd",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-12-10 23:40:47 +03:00
|
|
|
}
|
|
|
|
|
/* unpack the name of the proc that terminated */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &proc, &n, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* unpack the termination status */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &status, &n, OPAL_INT32))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* pass it down for processing */
|
|
|
|
|
orte_base_default_waitpid_fired(&proc, status);
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
/**** IOF_COMPLETE COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_IOF_COMPLETE:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
|
|
|
|
opal_output(0, "%s orted_cmd: received iof_complete cmd",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-12-10 23:40:47 +03:00
|
|
|
}
|
|
|
|
|
/* unpack the name of the proc that completed */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &proc, &n, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* pass it down for processing */
|
|
|
|
|
orte_odls_base_notify_iof_complete(&proc);
|
|
|
|
|
break;
|
|
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
/**** EXIT COMMAND ****/
|
2009-05-11 18:11:44 +04:00
|
|
|
case ORTE_DAEMON_EXIT_CMD:
|
2008-11-20 21:55:59 +03:00
|
|
|
if (orte_debug_daemons_flag) {
|
2009-05-11 18:11:44 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received exit cmd",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-11-20 21:55:59 +03:00
|
|
|
}
|
2009-05-11 18:11:44 +04:00
|
|
|
/* if we are the HNP, just kill our local procs */
|
2009-05-04 15:07:40 +04:00
|
|
|
if (ORTE_PROC_IS_HNP) {
|
2008-06-03 18:23:04 +04:00
|
|
|
orte_odls.kill_local_procs(ORTE_JOBID_WILDCARD, false);
|
2007-07-23 22:36:33 +04:00
|
|
|
return ORTE_SUCCESS;
|
|
|
|
|
}
|
2009-05-11 18:11:44 +04:00
|
|
|
|
|
|
|
|
/* else we are a daemon, trigger our exit - we will kill our
|
|
|
|
|
* local procs on our way out
|
2008-11-01 00:10:00 +03:00
|
|
|
*/
|
2009-04-30 19:08:02 +04:00
|
|
|
orte_trigger_event(&orte_exit);
|
2008-11-01 00:10:00 +03:00
|
|
|
return ORTE_SUCCESS;
|
|
|
|
|
break;
|
|
|
|
|
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
/**** HALT VM COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_HALT_VM_CMD:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received halt vm",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
}
|
|
|
|
|
/* trigger our appropriate exit procedure
|
|
|
|
|
* NOTE: this event will fire -after- any zero-time events
|
|
|
|
|
* so any pending relays -do- get sent first
|
|
|
|
|
*/
|
2008-08-05 19:09:29 +04:00
|
|
|
orte_trigger_event(&orte_exit);
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
return ORTE_SUCCESS;
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
/**** SPAWN JOB COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_SPAWN_JOB_CMD:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received spawn job",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
}
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
job = ORTE_JOBID_INVALID;
|
|
|
|
|
/* can only process this if we are the HNP */
|
2009-05-04 15:07:40 +04:00
|
|
|
if (ORTE_PROC_IS_HNP) {
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
/* unpack the job data */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &jdata, &n, ORTE_JOB))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto ANSWER_LAUNCH;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* launch it */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_plm.spawn(jdata))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto ANSWER_LAUNCH;
|
|
|
|
|
}
|
|
|
|
|
job = jdata->jobid;
|
|
|
|
|
}
|
|
|
|
|
ANSWER_LAUNCH:
|
|
|
|
|
/* pack the jobid to be returned */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &job, 1, ORTE_JOBID))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* return response */
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, ORTE_RML_TAG_TOOL, 0)) {
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
OBJ_RELEASE(answer);
|
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation.
Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway.
So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked.
Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion.
This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
|
|
|
break;
|
|
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
/**** CONTACT QUERY COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_CONTACT_QUERY_CMD:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received contact query",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2007-07-23 22:36:33 +04:00
|
|
|
}
|
|
|
|
|
/* send back contact info */
|
|
|
|
|
contact_info = orte_rml.get_contact_info();
|
|
|
|
|
|
|
|
|
|
if (NULL == contact_info) {
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERROR);
|
|
|
|
|
ret = ORTE_ERROR;
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* setup buffer with answer */
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &contact_info, 1, OPAL_STRING))) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, tag, 0)) {
|
2007-07-23 22:36:33 +04:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
OBJ_RELEASE(answer);
|
2007-07-23 22:36:33 +04:00
|
|
|
break;
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/**** REPORT_JOB_INFO_CMD COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_REPORT_JOB_INFO_CMD:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received job info query",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
/* if we are not the HNP, we can do nothing - report
|
|
|
|
|
* back 0 procs so the tool won't hang
|
|
|
|
|
*/
|
2009-05-04 15:07:40 +04:00
|
|
|
if (!ORTE_PROC_IS_HNP) {
|
2009-05-15 17:21:18 +04:00
|
|
|
int32_t zero=0;
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
2009-05-15 17:21:18 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &zero, 1, OPAL_INT32))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* callback function will release buffer */
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, ORTE_RML_TAG_TOOL, 0)) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
/* if we are the HNP, process the request */
|
2009-05-15 17:21:18 +04:00
|
|
|
int32_t i, num_jobs;
|
|
|
|
|
orte_job_t *jobdat;
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
/* unpack the jobid */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &job, &n, ORTE_JOBID))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* setup return */
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
|
|
|
|
|
/* if they asked for a specific job, then just get that info */
|
|
|
|
|
if (ORTE_JOBID_WILDCARD != job) {
|
2009-05-15 17:21:18 +04:00
|
|
|
job = ORTE_CONSTRUCT_LOCAL_JOBID(ORTE_PROC_MY_NAME->jobid, job);
|
2008-02-28 04:57:57 +03:00
|
|
|
if (NULL != (jobdat = orte_get_job_data_object(job))) {
|
|
|
|
|
num_jobs = 1;
|
2009-05-15 17:21:18 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &num_jobs, 1, OPAL_INT32))) {
|
2009-03-03 16:38:29 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &jobdat, 1, ORTE_JOB))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
} else {
|
|
|
|
|
/* if we get here, then send a zero answer */
|
|
|
|
|
num_jobs = 0;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &num_jobs, 1, OPAL_INT32))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
} else {
|
2009-03-03 16:38:29 +03:00
|
|
|
/* since the job array is no longer
|
|
|
|
|
* left-justified and may have holes, we have
|
|
|
|
|
* to cnt the number of jobs
|
|
|
|
|
*/
|
2009-05-15 17:21:18 +04:00
|
|
|
num_jobs = 0;
|
|
|
|
|
for (i=1; i < orte_job_data->size; i++) {
|
|
|
|
|
if (NULL != opal_pointer_array_get_item(orte_job_data, i)) {
|
2009-03-03 16:38:29 +03:00
|
|
|
num_jobs++;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
/* pack the number of jobs */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &num_jobs, 1, OPAL_INT32))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* now pack the data, one at a time */
|
|
|
|
|
for (i=1; i < orte_job_data->size; i++) {
|
|
|
|
|
if (NULL != (jobdat = (orte_job_t*)opal_pointer_array_get_item(orte_job_data, i))) {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &jobdat, 1, ORTE_JOB))) {
|
2009-03-03 16:38:29 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, ORTE_RML_TAG_TOOL, 0)) {
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
2008-02-28 04:57:57 +03:00
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
OBJ_RELEASE(answer);
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
2007-07-23 22:36:33 +04:00
|
|
|
break;
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/**** REPORT_NODE_INFO_CMD COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_REPORT_NODE_INFO_CMD:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received node info query",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
/* if we are not the HNP, we can do nothing - report
|
|
|
|
|
* back 0 nodes so the tool won't hang
|
|
|
|
|
*/
|
2009-05-04 15:07:40 +04:00
|
|
|
if (!ORTE_PROC_IS_HNP) {
|
2009-05-15 17:21:18 +04:00
|
|
|
int32_t zero=0;
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
2009-05-15 17:21:18 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &zero, 1, OPAL_INT32))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, ORTE_RML_TAG_TOOL, 0)) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
OBJ_RELEASE(answer);
|
2008-02-28 04:57:57 +03:00
|
|
|
} else {
|
|
|
|
|
/* if we are the HNP, process the request */
|
2009-05-15 17:21:18 +04:00
|
|
|
int32_t i, num_nodes;
|
|
|
|
|
orte_node_t *node;
|
2008-04-30 23:49:53 +04:00
|
|
|
char *nid;
|
2008-02-28 04:57:57 +03:00
|
|
|
|
2008-04-30 23:49:53 +04:00
|
|
|
/* unpack the nodename */
|
2008-02-28 04:57:57 +03:00
|
|
|
n = 1;
|
2008-04-30 23:49:53 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &nid, &n, OPAL_STRING))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* setup return */
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
2009-05-18 18:21:29 +04:00
|
|
|
num_nodes = 0;
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
/* if they asked for a specific node, then just get that info */
|
2008-04-30 23:49:53 +04:00
|
|
|
if (NULL != nid) {
|
2008-02-28 04:57:57 +03:00
|
|
|
/* find this node */
|
|
|
|
|
for (i=0; i < orte_node_pool->size; i++) {
|
2009-05-15 17:21:18 +04:00
|
|
|
if (NULL == (node = (orte_node_t*)opal_pointer_array_get_item(orte_node_pool, i))) {
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
if (0 == strcmp(nid, node->name)) {
|
2008-02-28 04:57:57 +03:00
|
|
|
num_nodes = 1;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &num_nodes, 1, OPAL_INT32))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &node, 1, ORTE_NODE))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
} else {
|
|
|
|
|
/* count number of nodes */
|
|
|
|
|
for (i=0; i < orte_node_pool->size; i++) {
|
2009-05-15 17:21:18 +04:00
|
|
|
if (NULL != opal_pointer_array_get_item(orte_node_pool, i)) {
|
|
|
|
|
num_nodes++;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
/* pack the answer */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &num_nodes, 1, OPAL_INT32))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
/* pack each node separately */
|
|
|
|
|
for (i=0; i < orte_node_pool->size; i++) {
|
|
|
|
|
if (NULL != (node = (orte_node_t*)opal_pointer_array_get_item(orte_node_pool, i))) {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &node, 1, ORTE_NODE))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
/* send the info */
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, ORTE_RML_TAG_TOOL, 0)) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
OBJ_RELEASE(answer);
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
/**** REPORT_PROC_INFO_CMD COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_REPORT_PROC_INFO_CMD:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_cmd: received proc info query",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
/* if we are not the HNP, we can do nothing - report
|
|
|
|
|
* back 0 procs so the tool won't hang
|
|
|
|
|
*/
|
2009-05-04 15:07:40 +04:00
|
|
|
if (!ORTE_PROC_IS_HNP) {
|
2009-05-15 17:21:18 +04:00
|
|
|
int32_t zero=0;
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
2009-05-15 17:21:18 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &zero, 1, OPAL_INT32))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, ORTE_RML_TAG_TOOL, 0)) {
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
2008-02-28 04:57:57 +03:00
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
/* if we are the HNP, process the request */
|
|
|
|
|
orte_job_t *jdata;
|
2009-05-15 17:21:18 +04:00
|
|
|
orte_proc_t *proc;
|
|
|
|
|
orte_vpid_t vpid;
|
|
|
|
|
int32_t i, num_procs;
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
/* setup the answer */
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
|
|
|
|
|
/* unpack the jobid */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &job, &n, ORTE_JOBID))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* look up job data object */
|
2009-05-15 17:21:18 +04:00
|
|
|
job = ORTE_CONSTRUCT_LOCAL_JOBID(ORTE_PROC_MY_NAME->jobid, job);
|
2008-02-28 04:57:57 +03:00
|
|
|
if (NULL == (jdata = orte_get_job_data_object(job))) {
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
|
2009-05-15 17:21:18 +04:00
|
|
|
goto CLEANUP;
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* unpack the vpid */
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &vpid, &n, ORTE_VPID))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
2009-05-15 17:21:18 +04:00
|
|
|
goto CLEANUP;
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* if they asked for a specific proc, then just get that info */
|
|
|
|
|
if (ORTE_VPID_WILDCARD != vpid) {
|
|
|
|
|
/* find this proc */
|
|
|
|
|
for (i=0; i < jdata->procs->size; i++) {
|
2009-05-15 17:21:18 +04:00
|
|
|
if (NULL == (proc = (orte_proc_t*)opal_pointer_array_get_item(jdata->procs, i))) {
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
if (vpid == proc->name.vpid) {
|
2008-02-28 04:57:57 +03:00
|
|
|
num_procs = 1;
|
2009-05-15 17:21:18 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &num_procs, 1, OPAL_INT32))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &proc, 1, ORTE_PROC))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
} else {
|
2009-05-15 17:21:18 +04:00
|
|
|
/* count number of procs */
|
|
|
|
|
num_procs = 0;
|
|
|
|
|
for (i=0; i < jdata->procs->size; i++) {
|
|
|
|
|
if (NULL != opal_pointer_array_get_item(jdata->procs, i)) {
|
|
|
|
|
num_procs++;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
/* pack the answer */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &num_procs, 1, OPAL_INT32))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
2009-05-15 17:21:18 +04:00
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
/* pack each proc separately */
|
|
|
|
|
for (i=0; i < jdata->procs->size; i++) {
|
|
|
|
|
if (NULL != (proc = (orte_proc_t*)opal_pointer_array_get_item(jdata->procs, i))) {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(answer, &proc, 1, ORTE_PROC))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
/* send the info */
|
2009-02-25 05:43:22 +03:00
|
|
|
if (0 > orte_rml.send_buffer(sender, answer, ORTE_RML_TAG_TOOL, 0)) {
|
2009-05-15 17:21:18 +04:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
2008-02-28 04:57:57 +03:00
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
2009-05-15 17:21:18 +04:00
|
|
|
OBJ_RELEASE(answer);
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
2007-07-23 22:36:33 +04:00
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
/**** HEARTBEAT COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_HEARTBEAT_CMD:
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_NOT_IMPLEMENTED);
|
|
|
|
|
ret = ORTE_ERR_NOT_IMPLEMENTED;
|
|
|
|
|
break;
|
|
|
|
|
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
/**** SYNC FROM LOCAL PROC ****/
|
|
|
|
|
case ORTE_DAEMON_SYNC_BY_PROC:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_recv: received sync from local proc %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_NAME_PRINT(sender));
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
}
|
2008-04-30 23:49:53 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls.require_sync(sender, buffer, false))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
case ORTE_DAEMON_SYNC_WANT_NIDMAP:
|
|
|
|
|
if (orte_debug_daemons_flag) {
|
2008-06-09 18:53:58 +04:00
|
|
|
opal_output(0, "%s orted_recv: received sync+nidmap from local proc %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_NAME_PRINT(sender));
|
2008-04-30 23:49:53 +04:00
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls.require_sync(sender, buffer, true))) {
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto CLEANUP;
|
|
|
|
|
}
|
|
|
|
|
break;
|
2008-12-22 23:23:05 +03:00
|
|
|
|
|
|
|
|
/**** TOP COMMAND ****/
|
|
|
|
|
case ORTE_DAEMON_TOP_CMD:
|
|
|
|
|
/* setup the answer */
|
|
|
|
|
answer = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
num_replies = 0;
|
|
|
|
|
hnp_accounted_for = false;
|
|
|
|
|
|
|
|
|
|
n = 1;
|
2009-02-09 23:39:05 +03:00
|
|
|
return_addr = NULL;
|
2008-12-22 23:23:05 +03:00
|
|
|
while (ORTE_SUCCESS == opal_dss.unpack(buffer, &proc, &n, ORTE_NAME)) {
|
|
|
|
|
/* the jobid provided will, of course, have the job family of
|
|
|
|
|
* the requestor. We need to convert that to our own job family
|
|
|
|
|
*/
|
|
|
|
|
proc.jobid = ORTE_CONSTRUCT_LOCAL_JOBID(ORTE_PROC_MY_NAME->jobid, proc.jobid);
|
2009-05-04 15:07:40 +04:00
|
|
|
if (ORTE_PROC_IS_HNP) {
|
2008-12-22 23:23:05 +03:00
|
|
|
return_addr = sender;
|
|
|
|
|
/* if the request is for a wildcard vpid, then it goes to every
|
|
|
|
|
* daemon. For scalability, we should probably xcast this some
|
|
|
|
|
* day - but for now, we just loop
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_VPID_WILDCARD == proc.vpid) {
|
|
|
|
|
/* loop across all daemons */
|
|
|
|
|
proc2.jobid = ORTE_PROC_MY_NAME->jobid;
|
2009-03-06 00:56:03 +03:00
|
|
|
for (proc2.vpid=1; proc2.vpid < orte_process_info.num_procs; proc2.vpid++) {
|
2008-12-22 23:23:05 +03:00
|
|
|
/* setup the cmd */
|
|
|
|
|
relay_msg = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
command = ORTE_DAEMON_TOP_CMD;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, &command, 1, ORTE_DAEMON_CMD))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, &proc, 1, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, sender, 1, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
/* the callback function will release relay_msg buffer */
|
2009-02-25 05:43:22 +03:00
|
|
|
#if 0
|
2008-12-22 23:23:05 +03:00
|
|
|
if (0 > orte_rml.send_buffer_nb(&proc2, relay_msg, ORTE_RML_TAG_DAEMON, 0,
|
|
|
|
|
send_callback, NULL)) {
|
2009-02-25 05:43:22 +03:00
|
|
|
#endif
|
|
|
|
|
if (0 > orte_rml.send_buffer(&proc2, relay_msg, ORTE_RML_TAG_DAEMON, 0)) {
|
2008-12-22 23:23:05 +03:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
|
|
|
|
num_replies++;
|
|
|
|
|
}
|
|
|
|
|
/* account for our own reply */
|
|
|
|
|
if (!hnp_accounted_for) {
|
|
|
|
|
hnp_accounted_for = true;
|
|
|
|
|
num_replies++;
|
|
|
|
|
}
|
|
|
|
|
/* now get the data for my own procs */
|
|
|
|
|
goto GET_TOP;
|
|
|
|
|
} else {
|
|
|
|
|
/* this is for a single proc - see which daemon
|
|
|
|
|
* this rank is on
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_VPID_INVALID == (proc2.vpid = orte_ess.proc_get_daemon(&proc))) {
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
/* if the vpid is me, then just handle this myself */
|
|
|
|
|
if (proc2.vpid == ORTE_PROC_MY_NAME->vpid) {
|
|
|
|
|
if (!hnp_accounted_for) {
|
|
|
|
|
hnp_accounted_for = true;
|
|
|
|
|
num_replies++;
|
|
|
|
|
}
|
|
|
|
|
goto GET_TOP;
|
|
|
|
|
}
|
|
|
|
|
/* otherwise, forward the cmd on to the appropriate daemon */
|
|
|
|
|
relay_msg = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
command = ORTE_DAEMON_TOP_CMD;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, &command, 1, ORTE_DAEMON_CMD))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
proc2.jobid = ORTE_PROC_MY_NAME->jobid;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, &proc, 1, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, sender, 1, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
/* the callback function will release relay_msg buffer */
|
2009-02-25 05:43:22 +03:00
|
|
|
#if 0
|
2008-12-22 23:23:05 +03:00
|
|
|
if (0 > orte_rml.send_buffer_nb(&proc2, relay_msg, ORTE_RML_TAG_DAEMON, 0,
|
|
|
|
|
send_callback, NULL)) {
|
2009-02-25 05:43:22 +03:00
|
|
|
#endif
|
|
|
|
|
if (0 > orte_rml.send_buffer(&proc2, relay_msg, ORTE_RML_TAG_DAEMON, 0)) {
|
2008-12-22 23:23:05 +03:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
OBJ_RELEASE(relay_msg);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
/* end if HNP */
|
|
|
|
|
} else {
|
|
|
|
|
/* this came from the HNP, but needs to go back to the original
|
|
|
|
|
* requestor. Unpack the name of that entity first
|
|
|
|
|
*/
|
|
|
|
|
n = 1;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.unpack(buffer, &proc2, &n, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
2009-01-05 18:35:02 +03:00
|
|
|
/* in this case, we are helpless - we have no idea who to send an
|
|
|
|
|
* error message TO! All we can do is return - the tool that sent
|
|
|
|
|
* this request is going to hang, but there isn't anything we can
|
|
|
|
|
* do about it
|
|
|
|
|
*/
|
|
|
|
|
goto CLEANUP;
|
2008-12-22 23:23:05 +03:00
|
|
|
}
|
|
|
|
|
return_addr = &proc2;
|
|
|
|
|
GET_TOP:
|
|
|
|
|
/* this rank must be local to me, or the HNP wouldn't
|
|
|
|
|
* have sent it to me - process the request
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls_base_get_proc_stats(answer, &proc))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
goto SEND_TOP_ANSWER;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
SEND_TOP_ANSWER:
|
|
|
|
|
/* send the answer back to requester - callback
|
|
|
|
|
* function will release buffer
|
|
|
|
|
*/
|
2009-05-04 15:07:40 +04:00
|
|
|
if (ORTE_PROC_IS_HNP) {
|
2008-12-22 23:23:05 +03:00
|
|
|
/* if I am the HNP, I need to also provide the number of
|
|
|
|
|
* replies the caller should recv and the sample time
|
|
|
|
|
*/
|
|
|
|
|
time_t mytime;
|
|
|
|
|
char *cptr;
|
|
|
|
|
|
|
|
|
|
relay_msg = OBJ_NEW(opal_buffer_t);
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, &num_replies, 1, OPAL_INT32))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
}
|
|
|
|
|
time(&mytime);
|
|
|
|
|
cptr = ctime(&mytime);
|
|
|
|
|
cptr[strlen(cptr)-1] = '\0'; /* remove trailing newline */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_dss.pack(relay_msg, &cptr, 1, OPAL_STRING))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
}
|
|
|
|
|
/* copy the stats payload */
|
|
|
|
|
opal_dss.copy_payload(relay_msg, answer);
|
|
|
|
|
OBJ_RELEASE(answer);
|
|
|
|
|
answer = relay_msg;
|
|
|
|
|
}
|
2009-02-09 23:39:05 +03:00
|
|
|
/* if we don't have a return address, then we are helpless */
|
|
|
|
|
if (NULL == return_addr) {
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
break;
|
|
|
|
|
}
|
2009-02-25 05:43:22 +03:00
|
|
|
#if 0
|
2008-12-22 23:23:05 +03:00
|
|
|
if (0 > orte_rml.send_buffer_nb(return_addr, answer, ORTE_RML_TAG_TOOL, 0,
|
|
|
|
|
send_callback, NULL)) {
|
2009-02-25 05:43:22 +03:00
|
|
|
#endif
|
|
|
|
|
if (0 > orte_rml.send_buffer(return_addr, answer, ORTE_RML_TAG_TOOL, 0)) {
|
2008-12-22 23:23:05 +03:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_COMM_FAILURE);
|
|
|
|
|
ret = ORTE_ERR_COMM_FAILURE;
|
|
|
|
|
}
|
|
|
|
|
break;
|
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.
This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:
As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.
In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.
The incoming changes revamp these procedures in three ways:
1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.
The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.
Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.
2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.
The size of this data has been reduced in three ways:
(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.
To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.
(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.
(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.
While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.
3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.
It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.
Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.
There are a few minor additional changes in the commit that I'll just note in passing:
* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.
* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.
* cleanup of some stale header files
This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
|
|
|
|
2007-07-23 22:36:33 +04:00
|
|
|
default:
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_BAD_PARAM);
|
|
|
|
|
ret = ORTE_ERR_BAD_PARAM;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
CLEANUP:
|
|
|
|
|
return ret;
|
|
|
|
|
}
|