2009-02-13 16:34:51 +00:00
|
|
|
/* -*- Mode: C; c-basic-offset:4 ; -*- */
|
2008-03-24 20:50:31 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
|
|
|
|
* University Research and Technology
|
|
|
|
|
* Corporation. All rights reserved.
|
2009-02-13 16:34:51 +00:00
|
|
|
* Copyright (c) 2004-2009 The University of Tennessee and The University
|
2008-03-24 20:50:31 +00:00
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
|
* reserved.
|
2009-02-10 17:20:13 +00:00
|
|
|
* Copyright (c) 2004-2009 High Performance Computing Center Stuttgart,
|
2008-03-24 20:50:31 +00:00
|
|
|
* University of Stuttgart. All rights reserved.
|
|
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
|
* All rights reserved.
|
|
|
|
|
* Copyright (c) 2007 Sun Microsystems, Inc. All rights reserved.
|
2011-10-19 20:18:14 +00:00
|
|
|
* Copyright (c) 2011 Cisco Systems, Inc. All rights reserved.
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
* Copyright (c) 2011-2013 Los Alamos National Security, LLC.
|
2012-04-06 14:23:13 +00:00
|
|
|
* All rights reserved.
|
2013-09-27 00:37:49 +00:00
|
|
|
* Copyright (c) 2013 Intel, Inc. All rights reserved.
|
2008-03-24 20:50:31 +00:00
|
|
|
* $COPYRIGHT$
|
|
|
|
|
*
|
|
|
|
|
* Additional copyrights may follow
|
|
|
|
|
*
|
|
|
|
|
* $HEADER$
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include "orte_config.h"
|
|
|
|
|
#include "orte/constants.h"
|
|
|
|
|
#include "orte/types.h"
|
|
|
|
|
|
|
|
|
|
#include <string.h>
|
|
|
|
|
#ifdef HAVE_SYS_TIME_H
|
|
|
|
|
#include <sys/time.h>
|
|
|
|
|
#endif /* HAVE_SYS_TIME_H */
|
|
|
|
|
|
|
|
|
|
#include "opal/threads/condition.h"
|
2009-02-14 02:26:12 +00:00
|
|
|
#include "opal/util/output.h"
|
2008-03-24 20:50:31 +00:00
|
|
|
#include "opal/class/opal_hash_table.h"
|
|
|
|
|
#include "opal/dss/dss.h"
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
#include "opal/mca/db/db.h"
|
2011-10-19 20:18:14 +00:00
|
|
|
#include "opal/mca/hwloc/base/base.h"
|
2009-02-06 15:25:06 +00:00
|
|
|
|
|
|
|
|
#include "orte/util/proc_info.h"
|
2008-03-24 20:50:31 +00:00
|
|
|
#include "orte/mca/errmgr/errmgr.h"
|
2009-02-06 15:25:06 +00:00
|
|
|
#include "orte/mca/ess/ess.h"
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 16:37:40 +00:00
|
|
|
#include "orte/mca/rml/rml.h"
|
2008-03-24 20:50:31 +00:00
|
|
|
#include "orte/runtime/orte_globals.h"
|
|
|
|
|
#include "orte/util/name_fns.h"
|
2009-02-06 15:25:06 +00:00
|
|
|
#include "orte/util/nidmap.h"
|
2008-03-24 20:50:31 +00:00
|
|
|
#include "orte/orted/orted.h"
|
|
|
|
|
#include "orte/runtime/orte_wait.h"
|
|
|
|
|
|
|
|
|
|
#include "orte/mca/grpcomm/base/base.h"
|
2009-02-06 15:25:06 +00:00
|
|
|
#include "orte/mca/grpcomm/grpcomm.h"
|
2008-03-24 20:50:31 +00:00
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
orte_grpcomm_coll_id_t orte_grpcomm_base_get_coll_id(void)
|
|
|
|
|
{
|
|
|
|
|
orte_grpcomm_coll_id_t id;
|
|
|
|
|
|
|
|
|
|
/* assign the next collective id */
|
|
|
|
|
id = orte_grpcomm_base.coll_id;
|
|
|
|
|
/* rotate to the next value */
|
|
|
|
|
orte_grpcomm_base.coll_id++;
|
|
|
|
|
return id;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2008-03-24 20:50:31 +00:00
|
|
|
/*************** MODEX SECTION **************/
|
2012-12-09 02:53:20 +00:00
|
|
|
void orte_grpcomm_base_modex(int fd, short args, void *cbdata)
|
2009-02-06 15:25:06 +00:00
|
|
|
{
|
2012-12-09 02:53:20 +00:00
|
|
|
orte_grpcomm_caddy_t *caddy = (orte_grpcomm_caddy_t*)cbdata;
|
|
|
|
|
orte_grpcomm_collective_t *modex = caddy->op;
|
2012-04-06 14:23:13 +00:00
|
|
|
int rc;
|
2012-09-12 12:03:24 +00:00
|
|
|
orte_namelist_t *nm;
|
|
|
|
|
opal_list_item_t *item;
|
2012-09-12 11:31:36 +00:00
|
|
|
bool found;
|
|
|
|
|
orte_grpcomm_collective_t *cptr;
|
2013-09-27 00:37:49 +00:00
|
|
|
opal_scope_t scope;
|
2011-10-19 20:18:14 +00:00
|
|
|
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_grpcomm_base_framework.framework_output,
|
2012-04-06 14:23:13 +00:00
|
|
|
"%s grpcomm:base:modex: performing modex",
|
2009-03-05 21:50:47 +00:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2009-02-06 15:25:06 +00:00
|
|
|
|
2013-10-04 02:58:26 +00:00
|
|
|
/* if we are a singleton and routing isn't enabled,
|
|
|
|
|
* then we have nobody with which to communicate, so
|
|
|
|
|
* we can just declare success
|
|
|
|
|
*/
|
|
|
|
|
if ((orte_process_info.proc_type & ORTE_PROC_SINGLETON) &&
|
|
|
|
|
!orte_routing_is_enabled) {
|
|
|
|
|
if (NULL != modex->cbfunc) {
|
|
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_grpcomm_base_framework.framework_output,
|
|
|
|
|
"%s CALLING MODEX RELEASE",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
|
|
|
|
modex->cbfunc(NULL, modex->cbdata);
|
|
|
|
|
}
|
|
|
|
|
/* flag the collective as complete */
|
|
|
|
|
modex->active = false;
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
if (0 == opal_list_get_size(&modex->participants)) {
|
2012-09-12 11:31:36 +00:00
|
|
|
/* record the collective */
|
|
|
|
|
modex->next_cbdata = modex;
|
|
|
|
|
opal_list_append(&orte_grpcomm_base.active_colls, &modex->super);
|
|
|
|
|
|
|
|
|
|
/* put our process name in the buffer so it can be unpacked later */
|
|
|
|
|
if (ORTE_SUCCESS != (rc = opal_dss.pack(&modex->buffer, ORTE_PROC_MY_NAME, 1, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
|
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
/* this is between our peers, so only collect info marked for them */
|
|
|
|
|
scope = OPAL_SCOPE_PEER;
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 16:37:40 +00:00
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
/* add a wildcard name to the participants so the daemon knows
|
2012-09-12 11:31:36 +00:00
|
|
|
* the jobid that is involved in this collective
|
2012-04-06 14:23:13 +00:00
|
|
|
*/
|
|
|
|
|
nm = OBJ_NEW(orte_namelist_t);
|
|
|
|
|
nm->name.jobid = ORTE_PROC_MY_NAME->jobid;
|
|
|
|
|
nm->name.vpid = ORTE_VPID_WILDCARD;
|
|
|
|
|
opal_list_append(&modex->participants, &nm->super);
|
|
|
|
|
modex->next_cb = orte_grpcomm_base_store_modex;
|
|
|
|
|
} else {
|
2012-09-12 11:31:36 +00:00
|
|
|
/* see if the collective is already present - a race condition
|
|
|
|
|
* exists where other participants may have already sent us their
|
|
|
|
|
* contribution. This would place the collective on the global
|
|
|
|
|
* array, but leave it marked as "inactive" until we call
|
|
|
|
|
* modex with the list of participants
|
|
|
|
|
*/
|
|
|
|
|
found = false;
|
|
|
|
|
for (item = opal_list_get_first(&orte_grpcomm_base.active_colls);
|
|
|
|
|
item != opal_list_get_end(&orte_grpcomm_base.active_colls);
|
|
|
|
|
item = opal_list_get_next(item)) {
|
|
|
|
|
cptr = (orte_grpcomm_collective_t*)item;
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((5, orte_grpcomm_base_framework.framework_output,
|
2012-09-12 11:31:36 +00:00
|
|
|
"%s CHECKING COLL id %d",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
cptr->id));
|
|
|
|
|
|
|
|
|
|
if (modex->id == cptr->id) {
|
|
|
|
|
found = true;
|
|
|
|
|
/* remove the old entry - we will replace it
|
|
|
|
|
* with the modex one
|
|
|
|
|
*/
|
|
|
|
|
opal_list_remove_item(&orte_grpcomm_base.active_colls, item);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
if (found) {
|
|
|
|
|
/* since it already exists, the list of
|
|
|
|
|
* targets contains the list of procs
|
|
|
|
|
* that have already sent us their info. Cycle
|
|
|
|
|
* thru the targets and move those entries to
|
|
|
|
|
* the modex object
|
|
|
|
|
*/
|
|
|
|
|
while (NULL != (item = opal_list_remove_first(&cptr->targets))) {
|
|
|
|
|
opal_list_append(&modex->targets, item);
|
|
|
|
|
}
|
|
|
|
|
/* copy the previously-saved data across */
|
|
|
|
|
opal_dss.copy_payload(&modex->local_bucket, &cptr->local_bucket);
|
|
|
|
|
/* cleanup */
|
|
|
|
|
OBJ_RELEASE(cptr);
|
|
|
|
|
}
|
|
|
|
|
/* now add the modex to the global list of active collectives */
|
2013-09-27 00:37:49 +00:00
|
|
|
modex->next_cb = orte_grpcomm_base_store_modex;
|
2012-09-12 11:31:36 +00:00
|
|
|
modex->next_cbdata = modex;
|
|
|
|
|
opal_list_append(&orte_grpcomm_base.active_colls, &modex->super);
|
|
|
|
|
|
|
|
|
|
/* pack the collective id */
|
|
|
|
|
if (ORTE_SUCCESS != (rc = opal_dss.pack(&modex->buffer, &modex->id, 1, ORTE_GRPCOMM_COLL_ID_T))) {
|
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* pack our name */
|
|
|
|
|
if (ORTE_SUCCESS != (rc = opal_dss.pack(&modex->buffer, ORTE_PROC_MY_NAME, 1, ORTE_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
2012-04-06 14:23:13 +00:00
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
/* this is not amongst our peers, but rather between a select
|
|
|
|
|
* group of processes - e.g., during a connect/accept operation.
|
|
|
|
|
* Thus, we need to include the non-peer info as well as our peers
|
|
|
|
|
* since we can't tell what the other participants may already have
|
|
|
|
|
*/
|
|
|
|
|
scope = OPAL_SCOPE_GLOBAL;
|
|
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
}
|
2011-10-19 20:18:14 +00:00
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
/* pack the requested entries */
|
|
|
|
|
if (ORTE_SUCCESS != (rc = orte_grpcomm_base_pack_modex_entries(&modex->buffer, scope))) {
|
2009-02-06 15:25:06 +00:00
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
|
|
|
|
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_grpcomm_base_framework.framework_output,
|
2009-02-06 15:25:06 +00:00
|
|
|
"%s grpcomm:base:full:modex: executing allgather",
|
2009-03-05 21:50:47 +00:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2009-02-06 15:25:06 +00:00
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
/* execute the allgather */
|
|
|
|
|
if (ORTE_SUCCESS != (rc = orte_grpcomm.allgather(modex))) {
|
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
2009-02-06 15:25:06 +00:00
|
|
|
}
|
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 03:40:11 +00:00
|
|
|
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_grpcomm_base_framework.framework_output,
|
2012-04-06 14:23:13 +00:00
|
|
|
"%s grpcomm:base:modex: modex posted",
|
2009-03-05 21:50:47 +00:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2009-02-06 15:25:06 +00:00
|
|
|
|
2012-12-09 02:53:20 +00:00
|
|
|
return;
|
2009-03-31 15:41:55 +00:00
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
cleanup:
|
2012-12-09 02:53:20 +00:00
|
|
|
return;
|
2012-04-06 14:23:13 +00:00
|
|
|
}
|
|
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
void orte_grpcomm_base_store_modex(opal_buffer_t *rbuf, void *cbdata)
|
2012-04-06 14:23:13 +00:00
|
|
|
{
|
2013-09-27 00:37:49 +00:00
|
|
|
int rc, j, cnt;
|
|
|
|
|
int32_t num_recvd_entries;
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
orte_process_name_t pname;
|
2012-04-06 14:23:13 +00:00
|
|
|
orte_grpcomm_collective_t *modex = (orte_grpcomm_collective_t*)cbdata;
|
|
|
|
|
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_grpcomm_base_framework.framework_output,
|
2013-09-27 00:37:49 +00:00
|
|
|
"%s STORING MODEX DATA",
|
2012-04-06 14:23:13 +00:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2009-02-25 02:43:22 +00:00
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
/* unpack the process name */
|
2009-02-06 15:25:06 +00:00
|
|
|
cnt=1;
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
while (ORTE_SUCCESS == (rc = opal_dss.unpack(rbuf, &pname, &cnt, ORTE_NAME))) {
|
2013-09-27 00:37:49 +00:00
|
|
|
/* unpack the number of entries for this proc */
|
|
|
|
|
cnt=1;
|
|
|
|
|
if (ORTE_SUCCESS != (rc = opal_dss.unpack(rbuf, &num_recvd_entries, &cnt, OPAL_INT32))) {
|
2012-06-27 14:53:55 +00:00
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
2009-02-06 15:25:06 +00:00
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((5, orte_grpcomm_base_framework.framework_output,
|
|
|
|
|
"%s grpcomm:base:update_modex_entries: adding %d entries for proc %s",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), num_recvd_entries,
|
|
|
|
|
ORTE_NAME_PRINT(&pname)));
|
2009-02-06 15:25:06 +00:00
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
/*
|
|
|
|
|
* Extract the attribute names and values
|
|
|
|
|
*/
|
|
|
|
|
for (j = 0; j < num_recvd_entries; j++) {
|
|
|
|
|
opal_value_t *kv;
|
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 03:40:11 +00:00
|
|
|
cnt = 1;
|
2013-09-27 00:37:49 +00:00
|
|
|
if (ORTE_SUCCESS != (rc = opal_dss.unpack(rbuf, &kv, &cnt, OPAL_VALUE))) {
|
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 03:40:11 +00:00
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
For large scale systems, we would like to avoid doing a full modex during MPI_Init so that launch will scale a little better. At the moment, our options are somewhat limited as only a few BTLs don't immediately call modex_recv on all procs during startup. However, for those situations where someone can take advantage of it, add the ability to do a "modex on demand" retrieval of data from remote procs when we launch via mpirun.
NOTE: launch performance will be absolutely awful if you do this with BTLs that aren't configured to modex_recv on first message!
Even with "modex on demand", we still have to do a barrier in place of the modex - we simply don't move any data around, which does reduce the time impact. The barrier is required to ensure that the other proc has in fact registered all its BTL info and therefore is prepared to hand over a complete data package. Otherwise, you may not get the info you need. In addition, the shared memory BTL can fail to properly rendezvous as it expects the barrier to be in place.
This behavior will *only* take effect under the following conditions:
1. launched via mpirun
2. #procs is greater than ompi_hostname_cutoff, which defaults to UINT32_MAX
3. mca param rte_orte_direct_modex is set to 1. At the moment, we are having problems getting this param to register properly, so only the first two conditions are in effect. Still, the bottom line is you have to *want* this behavior to get it.
The planned next evolution of this will be to make the direct modex be non-blocking - this will require two fixes:
1. if the remote proc doesn't have the required info, then let it delay its response until it does. This means we need a way for the MPI layer to tell the RTE "I am done entering modex data".
2. adjust the SM rendezvous logic to loop until the required file has been created
Creating a placeholder to bring this over to 1.7.5 when ready.
cmr=v1.7.5:reviewer=hjelmn:subject=Enable direct modex at scale
This commit was SVN r30259.
2014-01-11 17:36:06 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((10, orte_grpcomm_base_framework.framework_output,
|
|
|
|
|
"%s STORING MODEX DATA FROM %s FOR %s",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_NAME_PRINT(&pname), kv->key));
|
2013-09-27 00:37:49 +00:00
|
|
|
/* if this is me, dump the data - we already have it in the db */
|
|
|
|
|
if (ORTE_PROC_MY_NAME->jobid == pname.jobid &&
|
|
|
|
|
ORTE_PROC_MY_NAME->vpid == pname.vpid) {
|
|
|
|
|
OBJ_RELEASE(kv);
|
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 03:40:11 +00:00
|
|
|
} else {
|
2013-09-27 00:37:49 +00:00
|
|
|
/* store it in the database */
|
|
|
|
|
if (ORTE_SUCCESS != (rc = opal_db.store_pointer((opal_identifier_t*)&pname, kv))) {
|
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
|
|
|
|
/* do not release the kv - the db holds that pointer */
|
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 03:40:11 +00:00
|
|
|
}
|
|
|
|
|
}
|
2011-10-19 20:18:14 +00:00
|
|
|
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((5, orte_grpcomm_base_framework.framework_output,
|
2012-09-12 11:31:36 +00:00
|
|
|
"%s store:peer:modex: completed modex entry for proc %s",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
ORTE_NAME_PRINT(&pname)));
|
2013-09-27 00:37:49 +00:00
|
|
|
}
|
2012-04-06 14:23:13 +00:00
|
|
|
|
2011-05-17 03:30:25 +00:00
|
|
|
cleanup:
|
2013-08-23 18:02:50 +00:00
|
|
|
if (NULL == cbdata) {
|
|
|
|
|
return;
|
|
|
|
|
}
|
2012-04-09 22:44:56 +00:00
|
|
|
/* cleanup the list, but don't release the
|
|
|
|
|
* collective object as it was passed into us
|
|
|
|
|
*/
|
2012-04-06 14:23:13 +00:00
|
|
|
opal_list_remove_item(&orte_grpcomm_base.active_colls, &modex->super);
|
|
|
|
|
/* notify that the modex is complete */
|
|
|
|
|
if (NULL != modex->cbfunc) {
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_grpcomm_base_framework.framework_output,
|
2012-04-06 14:23:13 +00:00
|
|
|
"%s CALLING MODEX RELEASE",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
|
|
|
|
modex->cbfunc(NULL, modex->cbdata);
|
2012-09-12 11:31:36 +00:00
|
|
|
} else {
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_grpcomm_base_framework.framework_output,
|
2012-09-12 11:31:36 +00:00
|
|
|
"%s store:peer:modex NO MODEX RELEASE CBFUNC",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
2012-04-06 14:23:13 +00:00
|
|
|
}
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 16:37:40 +00:00
|
|
|
/* flag the collective as complete */
|
|
|
|
|
modex->active = false;
|
2009-02-06 15:25:06 +00:00
|
|
|
}
|
|
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
int orte_grpcomm_base_pack_modex_entries(opal_buffer_t *buf, opal_scope_t scope)
|
2008-06-18 22:17:53 +00:00
|
|
|
{
|
|
|
|
|
int rc;
|
2012-06-19 13:38:42 +00:00
|
|
|
int32_t num_entries;
|
2012-06-27 14:53:55 +00:00
|
|
|
opal_value_t *kv;
|
2012-06-19 13:38:42 +00:00
|
|
|
opal_list_t data;
|
|
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
/* fetch any global or local data */
|
2012-06-19 13:38:42 +00:00
|
|
|
OBJ_CONSTRUCT(&data, opal_list_t);
|
2013-09-27 00:37:49 +00:00
|
|
|
if (ORTE_SUCCESS != (rc = opal_db.fetch_multiple((opal_identifier_t*)ORTE_PROC_MY_NAME,
|
|
|
|
|
scope, NULL, &data))) {
|
2008-03-24 20:50:31 +00:00
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
2013-09-27 00:37:49 +00:00
|
|
|
num_entries = opal_list_get_size(&data);
|
2008-03-24 20:50:31 +00:00
|
|
|
|
2013-04-24 23:44:02 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((5, orte_grpcomm_base_framework.framework_output,
|
2012-06-19 13:38:42 +00:00
|
|
|
"%s grpcomm:base:pack_modex: reporting %d entries",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), num_entries));
|
2008-06-18 22:17:53 +00:00
|
|
|
|
2012-06-19 13:38:42 +00:00
|
|
|
/* put the number of entries into the buffer */
|
|
|
|
|
if (ORTE_SUCCESS != (rc = opal_dss.pack(buf, &num_entries, 1, OPAL_INT32))) {
|
2008-03-24 20:50:31 +00:00
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
|
goto cleanup;
|
|
|
|
|
}
|
|
|
|
|
|
2012-06-19 13:38:42 +00:00
|
|
|
/* if there are entries, store them */
|
2012-06-27 14:53:55 +00:00
|
|
|
while (NULL != (kv = (opal_value_t*)opal_list_remove_first(&data))) {
|
For large scale systems, we would like to avoid doing a full modex during MPI_Init so that launch will scale a little better. At the moment, our options are somewhat limited as only a few BTLs don't immediately call modex_recv on all procs during startup. However, for those situations where someone can take advantage of it, add the ability to do a "modex on demand" retrieval of data from remote procs when we launch via mpirun.
NOTE: launch performance will be absolutely awful if you do this with BTLs that aren't configured to modex_recv on first message!
Even with "modex on demand", we still have to do a barrier in place of the modex - we simply don't move any data around, which does reduce the time impact. The barrier is required to ensure that the other proc has in fact registered all its BTL info and therefore is prepared to hand over a complete data package. Otherwise, you may not get the info you need. In addition, the shared memory BTL can fail to properly rendezvous as it expects the barrier to be in place.
This behavior will *only* take effect under the following conditions:
1. launched via mpirun
2. #procs is greater than ompi_hostname_cutoff, which defaults to UINT32_MAX
3. mca param rte_orte_direct_modex is set to 1. At the moment, we are having problems getting this param to register properly, so only the first two conditions are in effect. Still, the bottom line is you have to *want* this behavior to get it.
The planned next evolution of this will be to make the direct modex be non-blocking - this will require two fixes:
1. if the remote proc doesn't have the required info, then let it delay its response until it does. This means we need a way for the MPI layer to tell the RTE "I am done entering modex data".
2. adjust the SM rendezvous logic to loop until the required file has been created
Creating a placeholder to bring this over to 1.7.5 when ready.
cmr=v1.7.5:reviewer=hjelmn:subject=Enable direct modex at scale
This commit was SVN r30259.
2014-01-11 17:36:06 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((5, orte_grpcomm_base_framework.framework_output,
|
|
|
|
|
"%s grpcomm:base:pack_modex: packing entry for %s",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), kv->key));
|
2013-09-17 23:50:11 +00:00
|
|
|
if (ORTE_SUCCESS != (rc = opal_dss.pack(buf, &kv, 1, OPAL_VALUE))) {
|
2008-06-18 22:17:53 +00:00
|
|
|
ORTE_ERROR_LOG(rc);
|
2012-06-19 13:38:42 +00:00
|
|
|
break;
|
2008-03-24 20:50:31 +00:00
|
|
|
}
|
2012-06-19 13:38:42 +00:00
|
|
|
OBJ_RELEASE(kv);
|
2008-03-24 20:50:31 +00:00
|
|
|
}
|
2012-06-16 09:11:03 +00:00
|
|
|
|
2012-06-19 13:38:42 +00:00
|
|
|
cleanup:
|
2012-06-27 14:53:55 +00:00
|
|
|
while (NULL != (kv = (opal_value_t*)opal_list_remove_first(&data))) {
|
2012-06-19 13:38:42 +00:00
|
|
|
OBJ_RELEASE(kv);
|
2012-06-16 09:11:03 +00:00
|
|
|
}
|
2012-06-19 13:38:42 +00:00
|
|
|
OBJ_DESTRUCT(&data);
|
|
|
|
|
|
2012-06-16 09:11:03 +00:00
|
|
|
return rc;
|
|
|
|
|
}
|