2011-10-17 20:51:22 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
|
|
|
|
* University Research and Technology
|
|
|
|
|
* Corporation. All rights reserved.
|
|
|
|
|
* Copyright (c) 2004-2011 The University of Tennessee and The University
|
|
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
|
* reserved.
|
|
|
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
|
|
|
* University of Stuttgart. All rights reserved.
|
|
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
|
* All rights reserved.
|
Per RFC, bring in the following changes:
* Remove paffinity, maffinity, and carto frameworks -- they've been
wholly replaced by hwloc.
* Move ompi_mpi_init() affinity-setting/checking code down to ORTE.
* Update sm, smcuda, wv, and openib components to no longer use carto.
Instead, use hwloc data. There are still optimizations possible in
the sm/smcuda BTLs (i.e., making multiple mpools). Also, the old
carto-based code found out how many NUMA nodes were ''available''
-- not how many were used ''in this job''. The new hwloc-using
code computes the same value -- it was not updated to calculate how
many NUMA nodes are used ''by this job.''
* Note that I cannot compile the smcuda and wv BTLs -- I ''think''
they're right, but they need to be verified by their owners.
* The openib component now does a bunch of stuff to figure out where
"near" OpenFabrics devices are. '''THIS IS A CHANGE IN DEFAULT
BEHAVIOR!!''' and still needs to be verified by OpenFabrics vendors
(I do not have a NUMA machine with an OpenFabrics device that is a
non-uniform distance from multiple different NUMA nodes).
* Completely rewrite the OMPI_Affinity_str() routine from the
"affinity" mpiext extension. This extension now understands
hyperthreads; the output format of it has changed a bit to reflect
this new information.
* Bunches of minor changes around the code base to update names/types
from maffinity/paffinity-based names to hwloc-based names.
* Add some helper functions into the hwloc base, mainly having to do
with the fact that we have the hwloc data reporting ''all''
topology information, but sometimes you really only want the
(online | available) data.
This commit was SVN r26391.
2012-05-07 14:52:54 +00:00
|
|
|
* Copyright (c) 2008-2012 Cisco Systems, Inc. All rights reserved.
|
2013-01-28 20:14:51 +00:00
|
|
|
* Copyright (c) 2012-2013 Los Alamos National Security, LLC.
|
2012-01-18 20:56:53 +00:00
|
|
|
* All rights reserved.
|
2013-07-18 14:46:07 +00:00
|
|
|
* Copyright (c) 2013 Intel, Inc. All rights reserved.
|
2011-10-17 20:51:22 +00:00
|
|
|
* $COPYRIGHT$
|
|
|
|
|
*
|
|
|
|
|
* Additional copyrights may follow
|
|
|
|
|
*
|
|
|
|
|
* $HEADER$
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include "orte_config.h"
|
|
|
|
|
#include "orte/constants.h"
|
|
|
|
|
|
|
|
|
|
#ifdef HAVE_UNISTD_H
|
|
|
|
|
#include <unistd.h>
|
|
|
|
|
#endif /* HAVE_UNISTD_H */
|
|
|
|
|
#ifdef HAVE_STRING_H
|
|
|
|
|
#include <string.h>
|
|
|
|
|
#endif /* HAVE_STRING_H */
|
|
|
|
|
#include <ctype.h>
|
|
|
|
|
#ifdef HAVE_NETDB_H
|
|
|
|
|
#include <netdb.h>
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef HAVE_IFADDRS_H
|
|
|
|
|
#include <ifaddrs.h>
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#include <pmi.h>
|
2013-07-18 22:45:08 +00:00
|
|
|
#if WANT_PMI2_SUPPORT
|
2013-07-18 14:46:07 +00:00
|
|
|
#include <pmi2.h>
|
|
|
|
|
#endif
|
2011-10-17 20:51:22 +00:00
|
|
|
|
|
|
|
|
#include "opal/util/opal_environ.h"
|
|
|
|
|
#include "opal/util/output.h"
|
|
|
|
|
#include "opal/util/argv.h"
|
|
|
|
|
#include "opal/class/opal_pointer_array.h"
|
2011-11-02 17:42:06 +00:00
|
|
|
#include "opal/mca/hwloc/base/base.h"
|
2011-10-17 20:51:22 +00:00
|
|
|
#include "opal/util/printf.h"
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
#include "opal/mca/common/pmi/common_pmi.h"
|
2011-10-17 20:51:22 +00:00
|
|
|
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
#include "opal/mca/db/db.h"
|
2012-06-27 14:53:55 +00:00
|
|
|
#include "orte/mca/errmgr/errmgr.h"
|
2013-10-08 18:37:59 +00:00
|
|
|
#include "orte/mca/grpcomm/grpcomm.h"
|
2013-01-03 02:16:10 +00:00
|
|
|
#include "orte/mca/rml/rml.h"
|
2011-10-17 20:51:22 +00:00
|
|
|
#include "orte/util/proc_info.h"
|
|
|
|
|
#include "orte/util/show_help.h"
|
|
|
|
|
#include "orte/util/name_fns.h"
|
|
|
|
|
#include "orte/util/nidmap.h"
|
|
|
|
|
#include "orte/util/pre_condition_transports.h"
|
|
|
|
|
#include "orte/util/regex.h"
|
2012-06-27 14:53:55 +00:00
|
|
|
#include "orte/runtime/orte_globals.h"
|
2011-10-17 20:51:22 +00:00
|
|
|
#include "orte/runtime/orte_wait.h"
|
|
|
|
|
|
|
|
|
|
#include "orte/mca/ess/ess.h"
|
|
|
|
|
#include "orte/mca/ess/base/base.h"
|
|
|
|
|
#include "orte/mca/ess/pmi/ess_pmi.h"
|
|
|
|
|
|
|
|
|
|
static int rte_init(void);
|
|
|
|
|
static int rte_finalize(void);
|
2012-04-06 14:23:13 +00:00
|
|
|
static void rte_abort(int error_code, bool report);
|
2011-10-17 20:51:22 +00:00
|
|
|
|
|
|
|
|
orte_ess_base_module_t orte_ess_pmi_module = {
|
|
|
|
|
rte_init,
|
|
|
|
|
rte_finalize,
|
|
|
|
|
rte_abort,
|
|
|
|
|
NULL /* ft_event */
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
static bool app_init_complete=false;
|
|
|
|
|
|
|
|
|
|
/**** MODULE FUNCTIONS ****/
|
|
|
|
|
|
|
|
|
|
static int rte_init(void)
|
|
|
|
|
{
|
2013-10-02 01:08:51 +00:00
|
|
|
int ret, i, j, procs;
|
When we direct launch an application, we rely on PMI for wireup support. In doing so, we lose the de facto data compression we get from the ORTE modex since we no longer get all the wireup info from every proc in a single blob. Instead, we have to iterate over all the procs, calling PMI_KVS_get for every value we require.
This creates a really bad scaling behavior. Users have found a nearly 20% launch time differential between mpirun and PMI, with PMI being the slower method. Some of the problem is attributable to poor exchange algorithms in RM's like Slurm and Alps, but we make things worse by calling "get" so many times.
Nathan (with a tad advice from me) has attempted to alleviate this problem by reducing the number of "get" calls. This required the following changes:
* upon first request for data, have the OPAL db pmi component fetch and decode *all* the info from a given remote proc. It turned out we weren't caching the info, so we would continually request it and only decode the piece we needed for the immediate request. We now decode all the info and push it into the db hash component for local storage - and then all subsequent retrievals are fulfilled locally
* reduced the amount of data by eliminating the exchange of the OMPI_ARCH value if heterogeneity is not enabled. This was used solely as a check so we would error out if the system wasn't actually homogeneous, which was fine when we thought there was no cost in doing the check. Unfortunately, at large scale and with direct launch, there is a non-zero cost of making this test. We are open to finding a compromise (perhaps turning the test off if requested?), if people feel strongly about performing the test
* reduced the amount of RTE data being automatically fetched, and fetched the rest only upon request. In particular, we no longer immediately fetch the hostname (which is only used for error reporting), but instead get it when needed. Likewise for the RML uri as that info is only required for some (not all) environments. In addition, we no longer fetch the locality unless required, relying instead on the PMI clique info to tell us who is on our local node (if additional info is required, the fetch is performed when a modex_recv is issued).
Again, all this only impacts direct launch - all the info is provided when launched via mpirun as there is no added cost to getting it
Barring objections, we may move this (plus any required other pieces) to the 1.7 branch once it soaks for an appropriate time.
This commit was SVN r29040.
2013-08-17 00:49:18 +00:00
|
|
|
char *error = NULL, *localj;
|
2011-10-17 20:51:22 +00:00
|
|
|
int32_t jobfam, stepid;
|
2012-08-12 01:28:23 +00:00
|
|
|
char *envar, *ev1, *ev2;
|
2011-10-17 20:51:22 +00:00
|
|
|
uint64_t unique_key[2];
|
|
|
|
|
char *cs_env, *string_key;
|
|
|
|
|
char *pmi_id=NULL;
|
2013-07-19 01:18:41 +00:00
|
|
|
int *ranks=NULL;
|
2012-01-18 20:56:53 +00:00
|
|
|
orte_jobid_t jobid;
|
2013-01-03 02:16:10 +00:00
|
|
|
char *rmluri;
|
2011-10-17 20:51:22 +00:00
|
|
|
|
|
|
|
|
/* run the prolog */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_ess_base_std_prolog())) {
|
|
|
|
|
error = "orte_ess_base_std_prolog";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2011-11-02 17:42:06 +00:00
|
|
|
#if OPAL_HAVE_HWLOC
|
|
|
|
|
/* get the topology */
|
|
|
|
|
if (NULL == opal_hwloc_topology) {
|
|
|
|
|
if (OPAL_SUCCESS != opal_hwloc_base_get_topology()) {
|
|
|
|
|
error = "topology discovery";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2012-01-18 20:56:53 +00:00
|
|
|
if (ORTE_PROC_IS_DAEMON) { /* I am a daemon, launched by mpirun */
|
|
|
|
|
/* we had to be given a jobid */
|
2013-03-27 21:09:41 +00:00
|
|
|
if (NULL == orte_ess_base_jobid) {
|
2012-01-18 20:56:53 +00:00
|
|
|
error = "missing jobid";
|
|
|
|
|
ret = ORTE_ERR_FATAL;
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-03-27 21:09:41 +00:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_util_convert_string_to_jobid(&jobid, orte_ess_base_jobid))) {
|
2012-01-18 20:56:53 +00:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "convert jobid";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
ORTE_PROC_MY_NAME->jobid = jobid;
|
|
|
|
|
/* get our rank from PMI */
|
2013-07-18 19:07:16 +00:00
|
|
|
if (!mca_common_pmi_rank(&i)) {
|
2012-01-18 20:56:53 +00:00
|
|
|
error = "could not get PMI rank";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
ORTE_PROC_MY_NAME->vpid = i + 1; /* compensate for orterun */
|
|
|
|
|
|
|
|
|
|
/* get the number of procs from PMI */
|
2013-07-18 19:07:16 +00:00
|
|
|
if (!mca_common_pmi_size(&i)) {
|
2012-01-18 20:56:53 +00:00
|
|
|
error = "could not get PMI universe size";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
orte_process_info.num_procs = i + 1; /* compensate for orterun */
|
|
|
|
|
|
2012-03-02 23:10:48 +00:00
|
|
|
/* complete setup */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_ess_base_orted_setup(NULL))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_ess_base_orted_setup";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2012-01-18 20:56:53 +00:00
|
|
|
} else { /* we are a direct-launched MPI process */
|
2013-07-18 14:46:07 +00:00
|
|
|
#if WANT_PMI2_SUPPORT
|
|
|
|
|
/* Get domain id */
|
|
|
|
|
pmi_id = (char*)malloc(PMI2_MAX_VALLEN);
|
|
|
|
|
if (PMI_SUCCESS != (ret = PMI2_Job_GetId(pmi_id, PMI2_MAX_VALLEN))) {
|
|
|
|
|
error = "PMI2_Job_GetId failed";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
#else
|
2013-07-18 17:19:13 +00:00
|
|
|
{
|
|
|
|
|
int pmi_maxlen;
|
|
|
|
|
|
|
|
|
|
/* get our PMI id length */
|
|
|
|
|
if (PMI_SUCCESS != (ret = PMI_Get_id_length_max(&pmi_maxlen))) {
|
|
|
|
|
error = "PMI_Get_id_length_max";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
pmi_id = (char*)malloc(pmi_maxlen);
|
|
|
|
|
if (PMI_SUCCESS != (ret = PMI_Get_kvs_domain_id(pmi_id, pmi_maxlen))) {
|
|
|
|
|
free(pmi_id);
|
|
|
|
|
error = "PMI_Get_kvs_domain_id";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2012-01-18 20:56:53 +00:00
|
|
|
}
|
2013-07-18 14:46:07 +00:00
|
|
|
#endif
|
2012-01-18 20:56:53 +00:00
|
|
|
/* PMI is very nice to us - the domain id is an integer followed
|
|
|
|
|
* by a '.', followed by essentially a stepid. The first integer
|
|
|
|
|
* defines an overall job number. The second integer is the number of
|
|
|
|
|
* individual jobs we have run within that allocation. So we translate
|
|
|
|
|
* this as the overall job number equating to our job family, and
|
|
|
|
|
* the individual number equating to our local jobid
|
|
|
|
|
*/
|
|
|
|
|
jobfam = strtol(pmi_id, &localj, 10);
|
|
|
|
|
if (NULL == localj) {
|
|
|
|
|
/* hmmm - no '.', so let's just use zero */
|
|
|
|
|
stepid = 0;
|
|
|
|
|
} else {
|
|
|
|
|
localj++; /* step over the '.' */
|
|
|
|
|
stepid = strtol(localj, NULL, 10) + 1; /* add one to avoid looking like a daemon */
|
|
|
|
|
}
|
2011-10-17 20:51:22 +00:00
|
|
|
free(pmi_id);
|
|
|
|
|
|
2012-01-18 20:56:53 +00:00
|
|
|
/* now build the jobid */
|
|
|
|
|
ORTE_PROC_MY_NAME->jobid = ORTE_CONSTRUCT_LOCAL_JOBID(jobfam << 16, stepid);
|
2011-10-17 20:51:22 +00:00
|
|
|
|
2012-01-18 20:56:53 +00:00
|
|
|
/* get our rank */
|
2013-07-18 14:46:07 +00:00
|
|
|
if (!mca_common_pmi_rank(&i)) {
|
2012-01-18 20:56:53 +00:00
|
|
|
error = "could not get PMI rank";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
ORTE_PROC_MY_NAME->vpid = i;
|
2011-10-17 20:51:22 +00:00
|
|
|
|
2012-01-18 20:56:53 +00:00
|
|
|
/* get the number of procs from PMI */
|
2013-07-18 14:46:07 +00:00
|
|
|
if (!mca_common_pmi_size(&i)) {
|
2012-01-18 20:56:53 +00:00
|
|
|
error = "could not get PMI universe size";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
orte_process_info.num_procs = i;
|
2012-08-12 01:28:23 +00:00
|
|
|
/* push into the environ for pickup in MPI layer for
|
|
|
|
|
* MPI-3 required info key
|
|
|
|
|
*/
|
|
|
|
|
asprintf(&ev1, "OMPI_MCA_orte_ess_num_procs=%d", i);
|
|
|
|
|
putenv(ev1);
|
|
|
|
|
asprintf(&ev2, "OMPI_APP_CTX_NUM_PROCS=%d", i);
|
|
|
|
|
putenv(ev2);
|
2011-10-17 20:51:22 +00:00
|
|
|
|
2012-01-18 20:56:53 +00:00
|
|
|
/* setup transport keys in case the MPI layer needs them -
|
|
|
|
|
* we can use the jobfam and stepid as unique keys
|
|
|
|
|
* because they are unique values assigned by the RM
|
|
|
|
|
*/
|
|
|
|
|
unique_key[0] = (uint64_t)jobfam;
|
|
|
|
|
unique_key[1] = (uint64_t)stepid;
|
|
|
|
|
if (NULL == (string_key = orte_pre_condition_transports_print(unique_key))) {
|
|
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_OUT_OF_RESOURCE);
|
|
|
|
|
return ORTE_ERR_OUT_OF_RESOURCE;
|
|
|
|
|
}
|
2013-03-27 21:09:41 +00:00
|
|
|
if (OPAL_SUCCESS != mca_base_var_env_name ("orte_precondition_transports", &cs_env)) {
|
2012-01-18 20:56:53 +00:00
|
|
|
ORTE_ERROR_LOG(ORTE_ERR_OUT_OF_RESOURCE);
|
|
|
|
|
return ORTE_ERR_OUT_OF_RESOURCE;
|
|
|
|
|
}
|
|
|
|
|
asprintf(&envar, "%s=%s", cs_env, string_key);
|
|
|
|
|
putenv(envar);
|
|
|
|
|
/* cannot free the envar as that messes up our environ */
|
|
|
|
|
free(cs_env);
|
|
|
|
|
free(string_key);
|
2011-10-17 20:51:22 +00:00
|
|
|
|
2012-04-06 14:23:13 +00:00
|
|
|
/* our app_context number can only be 0 as we don't support
|
|
|
|
|
* dynamic spawns
|
|
|
|
|
*/
|
|
|
|
|
orte_process_info.app_num = 0;
|
2011-10-17 20:51:22 +00:00
|
|
|
|
2012-01-18 20:56:53 +00:00
|
|
|
/* setup my daemon's name - arbitrary, since we don't route
|
|
|
|
|
* messages
|
|
|
|
|
*/
|
|
|
|
|
ORTE_PROC_MY_DAEMON->jobid = 0;
|
|
|
|
|
ORTE_PROC_MY_DAEMON->vpid = 0;
|
|
|
|
|
|
2012-06-27 14:53:55 +00:00
|
|
|
/* ensure we pick the correct critical components */
|
|
|
|
|
putenv("OMPI_MCA_grpcomm=pmi");
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
putenv("OMPI_MCA_db_pmi_store_priority=100");
|
2012-06-27 14:53:55 +00:00
|
|
|
putenv("OMPI_MCA_routed=direct");
|
|
|
|
|
|
|
|
|
|
/* now use the default procedure to finish my setup */
|
2013-10-02 19:03:46 +00:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_ess_base_app_setup(false))) {
|
2012-01-18 20:56:53 +00:00
|
|
|
ORTE_ERROR_LOG(ret);
|
2012-06-27 14:53:55 +00:00
|
|
|
error = "orte_ess_base_app_setup";
|
2012-01-18 20:56:53 +00:00
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2013-07-18 14:46:07 +00:00
|
|
|
#if WANT_PMI2_SUPPORT
|
|
|
|
|
{
|
|
|
|
|
/* get our local proc info to find our local rank */
|
|
|
|
|
char *pmapping = (char*)malloc(PMI2_MAX_VALLEN);
|
2013-10-02 01:08:51 +00:00
|
|
|
int found, sid, nodes, k;
|
2013-07-18 14:46:07 +00:00
|
|
|
orte_vpid_t n;
|
|
|
|
|
char *p;
|
|
|
|
|
ret = PMI2_Info_GetJobAttr("PMI_process_mapping", pmapping, PMI2_MAX_VALLEN, &found);
|
2013-07-19 03:04:00 +00:00
|
|
|
if (!found || PMI_SUCCESS != ret) { /* can't check PMI2_SUCCESS as some folks (i.e., Cray) don't define it */
|
2013-07-18 14:46:07 +00:00
|
|
|
error = "could not get PMI_process_mapping (PMI2_Info_GetJobAttr() failed)";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
i = 0; n = 0; procs = 0;
|
|
|
|
|
if (NULL != (p = strstr(pmapping, "(vector"))) {
|
|
|
|
|
while (NULL != (p = strstr(p+1, ",("))) {
|
|
|
|
|
if (3 == sscanf(p, ",(%d,%d,%d)", &sid, &nodes, &procs)) {
|
|
|
|
|
for (k = 0; k < nodes; k++) {
|
|
|
|
|
if ((ORTE_PROC_MY_NAME->vpid >= n) &&
|
|
|
|
|
(ORTE_PROC_MY_NAME->vpid < (n + procs))) {
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
n += procs;
|
|
|
|
|
}
|
2013-10-02 19:03:46 +00:00
|
|
|
} else {
|
|
|
|
|
procs = 0;
|
2013-07-18 14:46:07 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
free(pmapping);
|
|
|
|
|
|
2013-10-02 19:03:46 +00:00
|
|
|
if (0 < procs) {
|
2013-07-18 14:46:07 +00:00
|
|
|
ranks = (int*)malloc(procs * sizeof(int));
|
|
|
|
|
for (i=0; i < procs; i++) {
|
|
|
|
|
ranks[i] = n + i;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (NULL == ranks) {
|
|
|
|
|
error = "could not get PMI_process_mapping";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
#else
|
2012-01-18 20:56:53 +00:00
|
|
|
/* get our local proc info to find our local rank */
|
2013-10-02 00:10:36 +00:00
|
|
|
if (PMI_SUCCESS != (ret = PMI_Get_clique_size(&procs))) {
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
OPAL_PMI_ERROR(ret, "PMI_Get_clique_size");
|
2012-01-18 20:56:53 +00:00
|
|
|
error = "could not get PMI clique size";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2012-07-31 21:21:50 +00:00
|
|
|
/* now get the specific ranks */
|
2013-10-02 00:10:36 +00:00
|
|
|
ranks = (int*)calloc(procs, sizeof(int));
|
2013-09-27 00:37:49 +00:00
|
|
|
if (NULL == ranks) {
|
|
|
|
|
error = "could not get memory for local ranks";
|
|
|
|
|
ret = ORTE_ERR_OUT_OF_RESOURCE;
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-10-02 00:10:36 +00:00
|
|
|
if (PMI_SUCCESS != (ret = PMI_Get_clique_ranks(ranks, procs))) {
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
OPAL_PMI_ERROR(ret, "PMI_Get_clique_ranks");
|
2012-01-18 20:56:53 +00:00
|
|
|
error = "could not get clique ranks";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-07-18 14:46:07 +00:00
|
|
|
#endif
|
2013-07-19 04:34:58 +00:00
|
|
|
/* store the number of local peers - remember, we want the number
|
|
|
|
|
* of peers that share the node WITH ME, so we have to subtract
|
|
|
|
|
* ourselves from that number
|
|
|
|
|
*/
|
2013-10-02 00:10:36 +00:00
|
|
|
orte_process_info.num_local_peers = procs - 1;
|
2012-02-21 16:43:17 +00:00
|
|
|
/* The clique ranks are returned in rank order, so
|
|
|
|
|
* cycle thru the array and update the local/node
|
|
|
|
|
* rank info
|
|
|
|
|
*/
|
2013-10-02 00:10:36 +00:00
|
|
|
for (j=0; j < procs; j++) {
|
2012-06-27 14:53:55 +00:00
|
|
|
if (ranks[j] == (int)ORTE_PROC_MY_NAME->vpid) {
|
2013-09-27 00:37:49 +00:00
|
|
|
orte_process_info.my_local_rank = (orte_local_rank_t)j;
|
|
|
|
|
orte_process_info.my_node_rank = (orte_node_rank_t)j;
|
|
|
|
|
break;
|
2012-06-27 14:53:55 +00:00
|
|
|
}
|
2012-01-18 20:56:53 +00:00
|
|
|
}
|
|
|
|
|
free(ranks);
|
|
|
|
|
|
Per RFC, bring in the following changes:
* Remove paffinity, maffinity, and carto frameworks -- they've been
wholly replaced by hwloc.
* Move ompi_mpi_init() affinity-setting/checking code down to ORTE.
* Update sm, smcuda, wv, and openib components to no longer use carto.
Instead, use hwloc data. There are still optimizations possible in
the sm/smcuda BTLs (i.e., making multiple mpools). Also, the old
carto-based code found out how many NUMA nodes were ''available''
-- not how many were used ''in this job''. The new hwloc-using
code computes the same value -- it was not updated to calculate how
many NUMA nodes are used ''by this job.''
* Note that I cannot compile the smcuda and wv BTLs -- I ''think''
they're right, but they need to be verified by their owners.
* The openib component now does a bunch of stuff to figure out where
"near" OpenFabrics devices are. '''THIS IS A CHANGE IN DEFAULT
BEHAVIOR!!''' and still needs to be verified by OpenFabrics vendors
(I do not have a NUMA machine with an OpenFabrics device that is a
non-uniform distance from multiple different NUMA nodes).
* Completely rewrite the OMPI_Affinity_str() routine from the
"affinity" mpiext extension. This extension now understands
hyperthreads; the output format of it has changed a bit to reflect
this new information.
* Bunches of minor changes around the code base to update names/types
from maffinity/paffinity-based names to hwloc-based names.
* Add some helper functions into the hwloc base, mainly having to do
with the fact that we have the hwloc data reporting ''all''
topology information, but sometimes you really only want the
(online | available) data.
This commit was SVN r26391.
2012-05-07 14:52:54 +00:00
|
|
|
/* setup process binding */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_ess_base_proc_binding())) {
|
|
|
|
|
error = "proc_binding";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-08-09 22:39:41 +00:00
|
|
|
|
|
|
|
|
/* this needs to be set to enable debugger use when direct launched */
|
|
|
|
|
orte_standalone_operation = true;
|
2011-10-17 20:51:22 +00:00
|
|
|
}
|
2012-01-18 20:56:53 +00:00
|
|
|
|
|
|
|
|
/* set max procs */
|
|
|
|
|
if (orte_process_info.max_procs < orte_process_info.num_procs) {
|
|
|
|
|
orte_process_info.max_procs = orte_process_info.num_procs;
|
|
|
|
|
}
|
|
|
|
|
|
2013-01-03 02:16:10 +00:00
|
|
|
/* construct the PMI RTE string */
|
|
|
|
|
rmluri = orte_rml.get_contact_info();
|
When we direct launch an application, we rely on PMI for wireup support. In doing so, we lose the de facto data compression we get from the ORTE modex since we no longer get all the wireup info from every proc in a single blob. Instead, we have to iterate over all the procs, calling PMI_KVS_get for every value we require.
This creates a really bad scaling behavior. Users have found a nearly 20% launch time differential between mpirun and PMI, with PMI being the slower method. Some of the problem is attributable to poor exchange algorithms in RM's like Slurm and Alps, but we make things worse by calling "get" so many times.
Nathan (with a tad advice from me) has attempted to alleviate this problem by reducing the number of "get" calls. This required the following changes:
* upon first request for data, have the OPAL db pmi component fetch and decode *all* the info from a given remote proc. It turned out we weren't caching the info, so we would continually request it and only decode the piece we needed for the immediate request. We now decode all the info and push it into the db hash component for local storage - and then all subsequent retrievals are fulfilled locally
* reduced the amount of data by eliminating the exchange of the OMPI_ARCH value if heterogeneity is not enabled. This was used solely as a check so we would error out if the system wasn't actually homogeneous, which was fine when we thought there was no cost in doing the check. Unfortunately, at large scale and with direct launch, there is a non-zero cost of making this test. We are open to finding a compromise (perhaps turning the test off if requested?), if people feel strongly about performing the test
* reduced the amount of RTE data being automatically fetched, and fetched the rest only upon request. In particular, we no longer immediately fetch the hostname (which is only used for error reporting), but instead get it when needed. Likewise for the RML uri as that info is only required for some (not all) environments. In addition, we no longer fetch the locality unless required, relying instead on the PMI clique info to tell us who is on our local node (if additional info is required, the fetch is performed when a modex_recv is issued).
Again, all this only impacts direct launch - all the info is provided when launched via mpirun as there is no added cost to getting it
Barring objections, we may move this (plus any required other pieces) to the 1.7 branch once it soaks for an appropriate time.
This commit was SVN r29040.
2013-08-17 00:49:18 +00:00
|
|
|
|
2013-09-27 00:37:49 +00:00
|
|
|
/* store our info as marked for distribution to both our peers and non-peers
|
|
|
|
|
* as there is no daemons available for routed communication
|
|
|
|
|
*/
|
2013-07-18 14:46:07 +00:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_db.store((opal_identifier_t*)ORTE_PROC_MY_NAME,
|
2013-09-27 00:37:49 +00:00
|
|
|
OPAL_SCOPE_GLOBAL, ORTE_DB_RMLURI,
|
2013-07-18 14:46:07 +00:00
|
|
|
rmluri, OPAL_STRING))) {
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
error = "db store uri";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
free(rmluri);
|
2013-07-18 14:46:07 +00:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_db.store((opal_identifier_t*)ORTE_PROC_MY_NAME,
|
2013-09-27 00:37:49 +00:00
|
|
|
OPAL_SCOPE_GLOBAL, ORTE_DB_HOSTNAME,
|
2013-07-18 14:46:07 +00:00
|
|
|
orte_process_info.nodename, OPAL_STRING))) {
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
error = "db store hostname";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-07-18 14:46:07 +00:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_db.store((opal_identifier_t*)ORTE_PROC_MY_NAME,
|
2013-09-27 00:37:49 +00:00
|
|
|
OPAL_SCOPE_GLOBAL, OPAL_DB_CPUSET,
|
2013-07-18 14:46:07 +00:00
|
|
|
orte_process_info.cpuset, OPAL_STRING))) {
|
2013-03-26 18:27:50 +00:00
|
|
|
error = "db store cpuset";
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
goto error;
|
|
|
|
|
}
|
2013-07-18 14:46:07 +00:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_db.store((opal_identifier_t*)ORTE_PROC_MY_NAME,
|
2014-02-17 01:40:56 +00:00
|
|
|
OPAL_SCOPE_GLOBAL, OPAL_DB_LOCALRANK,
|
2013-07-18 14:46:07 +00:00
|
|
|
&orte_process_info.my_local_rank, ORTE_LOCAL_RANK))) {
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
error = "db store local rank";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-07-18 14:46:07 +00:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_db.store((opal_identifier_t*)ORTE_PROC_MY_NAME,
|
2013-09-27 00:37:49 +00:00
|
|
|
OPAL_SCOPE_GLOBAL, ORTE_DB_NODERANK,
|
2013-07-18 14:46:07 +00:00
|
|
|
&orte_process_info.my_node_rank, ORTE_NODE_RANK))) {
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 17:50:04 +00:00
|
|
|
error = "db store node rank";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
When we direct launch an application, we rely on PMI for wireup support. In doing so, we lose the de facto data compression we get from the ORTE modex since we no longer get all the wireup info from every proc in a single blob. Instead, we have to iterate over all the procs, calling PMI_KVS_get for every value we require.
This creates a really bad scaling behavior. Users have found a nearly 20% launch time differential between mpirun and PMI, with PMI being the slower method. Some of the problem is attributable to poor exchange algorithms in RM's like Slurm and Alps, but we make things worse by calling "get" so many times.
Nathan (with a tad advice from me) has attempted to alleviate this problem by reducing the number of "get" calls. This required the following changes:
* upon first request for data, have the OPAL db pmi component fetch and decode *all* the info from a given remote proc. It turned out we weren't caching the info, so we would continually request it and only decode the piece we needed for the immediate request. We now decode all the info and push it into the db hash component for local storage - and then all subsequent retrievals are fulfilled locally
* reduced the amount of data by eliminating the exchange of the OMPI_ARCH value if heterogeneity is not enabled. This was used solely as a check so we would error out if the system wasn't actually homogeneous, which was fine when we thought there was no cost in doing the check. Unfortunately, at large scale and with direct launch, there is a non-zero cost of making this test. We are open to finding a compromise (perhaps turning the test off if requested?), if people feel strongly about performing the test
* reduced the amount of RTE data being automatically fetched, and fetched the rest only upon request. In particular, we no longer immediately fetch the hostname (which is only used for error reporting), but instead get it when needed. Likewise for the RML uri as that info is only required for some (not all) environments. In addition, we no longer fetch the locality unless required, relying instead on the PMI clique info to tell us who is on our local node (if additional info is required, the fetch is performed when a modex_recv is issued).
Again, all this only impacts direct launch - all the info is provided when launched via mpirun as there is no added cost to getting it
Barring objections, we may move this (plus any required other pieces) to the 1.7 branch once it soaks for an appropriate time.
This commit was SVN r29040.
2013-08-17 00:49:18 +00:00
|
|
|
|
2013-10-08 18:37:59 +00:00
|
|
|
/* if we are an ORTE app - and not an MPI app - then
|
|
|
|
|
* we need to exchange our connection info here.
|
|
|
|
|
* MPI_Init has its own modex, so we don't need to do
|
|
|
|
|
* two of them. However, if we don't do a modex at all,
|
|
|
|
|
* then processes have no way to communicate
|
|
|
|
|
*
|
|
|
|
|
* NOTE: only do this when the process originally launches.
|
|
|
|
|
* Cannot do this on a restart as the rest of the processes
|
|
|
|
|
* in the job won't be executing this step, so we would hang
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_PROC_IS_NON_MPI && !orte_do_not_barrier) {
|
|
|
|
|
orte_grpcomm_collective_t coll;
|
|
|
|
|
OBJ_CONSTRUCT(&coll, orte_grpcomm_collective_t);
|
|
|
|
|
coll.id = orte_process_info.peer_modex;
|
|
|
|
|
coll.active = true;
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_grpcomm.modex(&coll))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte modex";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
ORTE_WAIT_FOR_COMPLETION(coll.active);
|
|
|
|
|
OBJ_DESTRUCT(&coll);
|
|
|
|
|
}
|
|
|
|
|
|
2011-10-17 20:51:22 +00:00
|
|
|
/* flag that we completed init */
|
|
|
|
|
app_init_complete = true;
|
|
|
|
|
|
|
|
|
|
return ORTE_SUCCESS;
|
|
|
|
|
|
2012-01-18 20:56:53 +00:00
|
|
|
error:
|
2011-11-22 21:24:35 +00:00
|
|
|
if (ORTE_ERR_SILENT != ret && !orte_report_silent_errors) {
|
|
|
|
|
orte_show_help("help-orte-runtime.txt",
|
|
|
|
|
"orte_init:startup:internal-failure",
|
|
|
|
|
true, error, ORTE_ERROR_NAME(ret), ret);
|
|
|
|
|
}
|
|
|
|
|
|
2011-10-17 20:51:22 +00:00
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static int rte_finalize(void)
|
|
|
|
|
{
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 16:37:40 +00:00
|
|
|
int ret;
|
|
|
|
|
|
2011-10-17 20:51:22 +00:00
|
|
|
if (app_init_complete) {
|
2012-01-18 20:56:53 +00:00
|
|
|
/* if I am a daemon, finalize using the default procedure */
|
|
|
|
|
if (ORTE_PROC_IS_DAEMON) {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_ess_base_orted_finalize())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 16:37:40 +00:00
|
|
|
return ret;
|
2012-01-18 20:56:53 +00:00
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
/* remove the envars that we pushed into environ
|
|
|
|
|
* so we leave that structure intact
|
|
|
|
|
*/
|
|
|
|
|
unsetenv("OMPI_MCA_grpcomm");
|
|
|
|
|
unsetenv("OMPI_MCA_routed");
|
2013-09-27 00:37:49 +00:00
|
|
|
unsetenv("OMPI_MCA_db_pmi_store_priority");
|
2012-01-18 20:56:53 +00:00
|
|
|
unsetenv("OMPI_MCA_orte_precondition_transports");
|
2013-09-27 00:37:49 +00:00
|
|
|
unsetenv("OMPI_MCA_orte_ess_num_procs");
|
|
|
|
|
unsetenv("OMPI_APP_CTX_NUM_PROCS");
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 16:37:40 +00:00
|
|
|
/* use the default app procedure to finish */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_ess_base_app_finalize())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
return ret;
|
|
|
|
|
}
|
2011-10-17 20:51:22 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* deconstruct my nidmap and jobmap arrays - this
|
|
|
|
|
* function protects itself from being called
|
|
|
|
|
* before things were initialized
|
|
|
|
|
*/
|
|
|
|
|
orte_util_nidmap_finalize();
|
|
|
|
|
|
2011-11-02 17:42:06 +00:00
|
|
|
#if OPAL_HAVE_HWLOC
|
|
|
|
|
if (NULL != opal_hwloc_topology) {
|
|
|
|
|
opal_hwloc_base_free_topology(opal_hwloc_topology);
|
|
|
|
|
opal_hwloc_topology = NULL;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 16:37:40 +00:00
|
|
|
return ORTE_SUCCESS;
|
2011-10-17 20:51:22 +00:00
|
|
|
}
|
|
|
|
|
|
2013-10-08 18:37:59 +00:00
|
|
|
static void rte_abort(int status, bool report)
|
2011-10-17 20:51:22 +00:00
|
|
|
{
|
2013-10-08 18:37:59 +00:00
|
|
|
OPAL_OUTPUT_VERBOSE((1, orte_ess_base_framework.framework_output,
|
|
|
|
|
"%s ess:pmi:abort: abort with status %d",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
status));
|
|
|
|
|
|
|
|
|
|
/* PMI doesn't like NULL messages, but our interface
|
|
|
|
|
* doesn't provide one - so rig one up here
|
|
|
|
|
*/
|
|
|
|
|
#if WANT_PMI2_SUPPORT
|
|
|
|
|
PMI2_Abort(status, "N/A");
|
|
|
|
|
#else
|
|
|
|
|
PMI_Abort(status, "N/A");
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/* - Clean out the global structures
|
|
|
|
|
* (not really necessary, but good practice) */
|
|
|
|
|
orte_proc_info_finalize();
|
|
|
|
|
|
|
|
|
|
/* Now Exit */
|
|
|
|
|
exit(status);
|
2011-10-17 20:51:22 +00:00
|
|
|
}
|