2008-02-28 04:57:57 +03:00
|
|
|
/*
|
2010-03-13 02:57:50 +03:00
|
|
|
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
2008-02-28 04:57:57 +03:00
|
|
|
* University Research and Technology
|
|
|
|
|
* Corporation. All rights reserved.
|
2011-06-24 00:38:02 +04:00
|
|
|
* Copyright (c) 2004-2011 The University of Tennessee and The University
|
2008-02-28 04:57:57 +03:00
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
|
* reserved.
|
|
|
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
|
|
|
* University of Stuttgart. All rights reserved.
|
|
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
|
* All rights reserved.
|
2011-06-15 17:10:13 +04:00
|
|
|
* Copyright (c) 2010-2011 Oak Ridge National Labs. All rights reserved.
|
Per RFC, bring in the following changes:
* Remove paffinity, maffinity, and carto frameworks -- they've been
wholly replaced by hwloc.
* Move ompi_mpi_init() affinity-setting/checking code down to ORTE.
* Update sm, smcuda, wv, and openib components to no longer use carto.
Instead, use hwloc data. There are still optimizations possible in
the sm/smcuda BTLs (i.e., making multiple mpools). Also, the old
carto-based code found out how many NUMA nodes were ''available''
-- not how many were used ''in this job''. The new hwloc-using
code computes the same value -- it was not updated to calculate how
many NUMA nodes are used ''by this job.''
* Note that I cannot compile the smcuda and wv BTLs -- I ''think''
they're right, but they need to be verified by their owners.
* The openib component now does a bunch of stuff to figure out where
"near" OpenFabrics devices are. '''THIS IS A CHANGE IN DEFAULT
BEHAVIOR!!''' and still needs to be verified by OpenFabrics vendors
(I do not have a NUMA machine with an OpenFabrics device that is a
non-uniform distance from multiple different NUMA nodes).
* Completely rewrite the OMPI_Affinity_str() routine from the
"affinity" mpiext extension. This extension now understands
hyperthreads; the output format of it has changed a bit to reflect
this new information.
* Bunches of minor changes around the code base to update names/types
from maffinity/paffinity-based names to hwloc-based names.
* Add some helper functions into the hwloc base, mainly having to do
with the fact that we have the hwloc data reporting ''all''
topology information, but sometimes you really only want the
(online | available) data.
This commit was SVN r26391.
2012-05-07 18:52:54 +04:00
|
|
|
* Copyright (c) 2011-2012 Cisco Systems, Inc. All rights reserved.
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 21:50:04 +04:00
|
|
|
* Copyright (c) 2011-2013 Los Alamos National Security, LLC. All rights
|
2012-04-06 18:23:13 +04:00
|
|
|
* reserved.
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
* Copyright (c) 2013 Intel, Inc. All rights reserved.
|
2008-02-28 04:57:57 +03:00
|
|
|
* $COPYRIGHT$
|
|
|
|
|
*
|
|
|
|
|
* Additional copyrights may follow
|
|
|
|
|
*
|
|
|
|
|
* $HEADER$
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include "orte_config.h"
|
|
|
|
|
#include "orte/constants.h"
|
|
|
|
|
|
|
|
|
|
#include <sys/types.h>
|
|
|
|
|
#include <stdio.h>
|
|
|
|
|
#ifdef HAVE_FCNTL_H
|
|
|
|
|
#include <fcntl.h>
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef HAVE_UNISTD_H
|
|
|
|
|
#include <unistd.h>
|
|
|
|
|
#endif
|
|
|
|
|
|
2010-03-23 23:47:41 +03:00
|
|
|
#include "opal/class/opal_list.h"
|
Per the meeting on moving the BTLs to OPAL, move the ORTE database "db" framework to OPAL so the relocated BTLs can access it. Because the data is indexed by process, this requires that we define a new "opal_identifier_t" that corresponds to the orte_process_name_t struct. In order to support multiple run-times, this is defined in opal/mca/db/db_types.h as a uint64_t without identifying the meaning of any part of that data.
A few changes were required to support this move:
1. the PMI component used to identify rte-related data (e.g., host name, bind level) and package them as a unit to reduce the number of PMI keys. This code was moved up to the ORTE layer as the OPAL layer has no understanding of these concepts. In addition, the component locally stored data based on process jobid/vpid - this could no longer be supported (see below for the solution).
2. the hash component was updated to use the new opal_identifier_t instead of orte_process_name_t as its index for storing data in the hash tables. Previously, we did a hash on the vpid and stored the data in a 32-bit hash table. In the revised system, we don't see a separate "vpid" field - we only have a 64-bit opaque value. The orte_process_name_t hash turned out to do nothing useful, so we now store the data in a 64-bit hash table. Preliminary tests didn't show any identifiable change in behavior or performance, but we'll have to see if a move back to the 32-bit table is required at some later time.
3. the db framework was a "select one" system. However, since the PMI component could no longer use its internal storage system, the framework has now been changed to a "select many" mode of operation. This allows the hash component to handle all internal storage, while the PMI component only handles pushing/pulling things from the PMI system. This was something we had planned for some time - when fetching data, we first check internal storage to see if we already have it, and then automatically go to the global system to look for it if we don't. Accordingly, the framework was provided with a custom query function used during "select" that lets you seperately specify the "store" and "fetch" ordering.
4. the ORTE grpcomm and ess/pmi components, and the nidmap code, were updated to work with the new db framework and to specify internal/global storage options.
No changes were made to the MPI layer, except for modifying the ORTE component of the OMPI/rte framework to support the new db framework.
This commit was SVN r28112.
2013-02-26 21:50:04 +04:00
|
|
|
#include "opal/mca/db/base/base.h"
|
Update libevent to the 2.0 series, currently at 2.0.7rc. We will update to their final release when it becomes available. Currently known errors exist in unused portions of the libevent code. This revision passes the IBM test suite on a Linux machine and on a standalone Mac.
This is a fairly intrusive change, but outside of the moving of opal/event to opal/mca/event, the only changes involved (a) changing all calls to opal_event functions to reflect the new framework instead, and (b) ensuring that all opal_event_t objects are properly constructed since they are now true opal_objects.
Note: Shiqing has just returned from vacation and has not yet had a chance to complete the Windows integration. Thus, this commit almost certainly breaks Windows support on the trunk. However, I want this to have a chance to soak for as long as possible before I become less available a week from today (going to be at a class for 5 days, and thus will only be sparingly available) so we can find and fix any problems.
Biggest change is moving the libevent code from opal/event to a new opal/mca/event framework. This was done to make it much easier to update libevent in the future. New versions can be inserted as a new component and tested in parallel with the current version until validated, then we can remove the earlier version if we so choose. This is a statically built framework ala installdirs, so only one component will build at a time. There is no selection logic - the sole compiled component simply loads its function pointers into the opal_event struct.
I have gone thru the code base and converted all the libevent calls I could find. However, I cannot compile nor test every environment. It is therefore quite likely that errors remain in the system. Please keep an eye open for two things:
1. compile-time errors: these will be obvious as calls to the old functions (e.g., opal_evtimer_new) must be replaced by the new framework APIs (e.g., opal_event.evtimer_new)
2. run-time errors: these will likely show up as segfaults due to missing constructors on opal_event_t objects. It appears that it became a typical practice for people to "init" an opal_event_t by simply using memset to zero it out. This will no longer work - you must either OBJ_NEW or OBJ_CONSTRUCT an opal_event_t. I tried to catch these cases, but may have missed some. Believe me, you'll know when you hit it.
There is also the issue of the new libevent "no recursion" behavior. As I described on a recent email, we will have to discuss this and figure out what, if anything, we need to do.
This commit was SVN r23925.
2010-10-24 22:35:54 +04:00
|
|
|
#include "opal/mca/event/event.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "opal/runtime/opal.h"
|
|
|
|
|
#include "opal/runtime/opal_cr.h"
|
|
|
|
|
|
2013-04-29 21:02:37 +04:00
|
|
|
#include "opal/util/argv.h"
|
2010-03-23 23:47:41 +03:00
|
|
|
#include "opal/util/if.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "opal/util/os_path.h"
|
2009-02-14 05:26:12 +03:00
|
|
|
#include "opal/util/output.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "opal/util/malloc.h"
|
2008-08-12 03:48:39 +04:00
|
|
|
#include "opal/util/basename.h"
|
2008-12-22 23:23:05 +03:00
|
|
|
#include "opal/mca/pstat/base/base.h"
|
2011-10-29 19:07:36 +04:00
|
|
|
#include "opal/mca/hwloc/base/base.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
#include "orte/mca/oob/base/base.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/rml/base/base.h"
|
2009-02-14 05:26:12 +03:00
|
|
|
#include "orte/mca/rml/rml_types.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/routed/base/base.h"
|
|
|
|
|
#include "orte/mca/routed/routed.h"
|
2012-10-25 21:15:17 +04:00
|
|
|
#include "orte/mca/dfs/base/base.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/errmgr/base/base.h"
|
|
|
|
|
#include "orte/mca/grpcomm/base/base.h"
|
|
|
|
|
#include "orte/mca/iof/base/base.h"
|
|
|
|
|
#include "orte/mca/ras/base/base.h"
|
|
|
|
|
#include "orte/mca/plm/base/base.h"
|
2010-07-18 01:03:27 +04:00
|
|
|
#include "orte/mca/plm/plm.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/odls/base/base.h"
|
2010-04-27 02:15:57 +04:00
|
|
|
#include "orte/mca/sensor/base/base.h"
|
2010-05-05 04:48:43 +04:00
|
|
|
#include "orte/mca/sensor/sensor.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/rmaps/base/base.h"
|
2010-03-13 02:57:50 +03:00
|
|
|
#if OPAL_ENABLE_FT_CR == 1
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/snapc/base/base.h"
|
|
|
|
|
#endif
|
|
|
|
|
#include "orte/mca/filem/base/base.h"
|
2012-04-06 18:23:13 +04:00
|
|
|
#include "orte/mca/state/base/base.h"
|
|
|
|
|
#include "orte/mca/state/state.h"
|
2010-07-18 01:03:27 +04:00
|
|
|
|
|
|
|
|
#include "orte/util/show_help.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/util/proc_info.h"
|
|
|
|
|
#include "orte/util/session_dir.h"
|
|
|
|
|
#include "orte/util/hnp_contact.h"
|
|
|
|
|
#include "orte/util/name_fns.h"
|
2008-06-09 18:53:58 +04:00
|
|
|
#include "orte/util/show_help.h"
|
2009-09-10 01:00:27 +04:00
|
|
|
#include "orte/util/comm/comm.h"
|
2011-06-24 00:38:02 +04:00
|
|
|
#include "orte/util/nidmap.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
#include "orte/runtime/runtime.h"
|
|
|
|
|
#include "orte/runtime/orte_wait.h"
|
|
|
|
|
#include "orte/runtime/orte_globals.h"
|
2010-07-18 01:03:27 +04:00
|
|
|
#include "orte/runtime/orte_quit.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/runtime/orte_cr.h"
|
2010-07-18 01:03:27 +04:00
|
|
|
#include "orte/runtime/orte_locks.h"
|
|
|
|
|
#include "orte/runtime/orte_data_server.h"
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/mca/ess/ess.h"
|
|
|
|
|
#include "orte/mca/ess/base/base.h"
|
|
|
|
|
#include "orte/mca/ess/hnp/ess_hnp.h"
|
|
|
|
|
|
2009-05-04 15:07:40 +04:00
|
|
|
static int rte_init(void);
|
2008-02-28 04:57:57 +03:00
|
|
|
static int rte_finalize(void);
|
|
|
|
|
static void rte_abort(int status, bool report) __opal_attribute_noreturn__;
|
|
|
|
|
|
|
|
|
|
orte_ess_base_module_t orte_ess_hnp_module = {
|
|
|
|
|
rte_init,
|
|
|
|
|
rte_finalize,
|
2008-03-05 07:57:23 +03:00
|
|
|
rte_abort,
|
|
|
|
|
NULL /* ft_event */
|
2008-02-28 04:57:57 +03:00
|
|
|
};
|
|
|
|
|
|
2010-07-18 01:03:27 +04:00
|
|
|
/* local globals */
|
|
|
|
|
static bool signals_set=false;
|
2012-04-06 18:23:13 +04:00
|
|
|
static bool forcibly_die=false;
|
Update libevent to the 2.0 series, currently at 2.0.7rc. We will update to their final release when it becomes available. Currently known errors exist in unused portions of the libevent code. This revision passes the IBM test suite on a Linux machine and on a standalone Mac.
This is a fairly intrusive change, but outside of the moving of opal/event to opal/mca/event, the only changes involved (a) changing all calls to opal_event functions to reflect the new framework instead, and (b) ensuring that all opal_event_t objects are properly constructed since they are now true opal_objects.
Note: Shiqing has just returned from vacation and has not yet had a chance to complete the Windows integration. Thus, this commit almost certainly breaks Windows support on the trunk. However, I want this to have a chance to soak for as long as possible before I become less available a week from today (going to be at a class for 5 days, and thus will only be sparingly available) so we can find and fix any problems.
Biggest change is moving the libevent code from opal/event to a new opal/mca/event framework. This was done to make it much easier to update libevent in the future. New versions can be inserted as a new component and tested in parallel with the current version until validated, then we can remove the earlier version if we so choose. This is a statically built framework ala installdirs, so only one component will build at a time. There is no selection logic - the sole compiled component simply loads its function pointers into the opal_event struct.
I have gone thru the code base and converted all the libevent calls I could find. However, I cannot compile nor test every environment. It is therefore quite likely that errors remain in the system. Please keep an eye open for two things:
1. compile-time errors: these will be obvious as calls to the old functions (e.g., opal_evtimer_new) must be replaced by the new framework APIs (e.g., opal_event.evtimer_new)
2. run-time errors: these will likely show up as segfaults due to missing constructors on opal_event_t objects. It appears that it became a typical practice for people to "init" an opal_event_t by simply using memset to zero it out. This will no longer work - you must either OBJ_NEW or OBJ_CONSTRUCT an opal_event_t. I tried to catch these cases, but may have missed some. Believe me, you'll know when you hit it.
There is also the issue of the new libevent "no recursion" behavior. As I described on a recent email, we will have to discuss this and figure out what, if anything, we need to do.
This commit was SVN r23925.
2010-10-24 22:35:54 +04:00
|
|
|
static opal_event_t term_handler;
|
|
|
|
|
static opal_event_t epipe_handler;
|
2012-08-03 01:15:35 +04:00
|
|
|
static int term_pipe[2];
|
Update libevent to the 2.0 series, currently at 2.0.7rc. We will update to their final release when it becomes available. Currently known errors exist in unused portions of the libevent code. This revision passes the IBM test suite on a Linux machine and on a standalone Mac.
This is a fairly intrusive change, but outside of the moving of opal/event to opal/mca/event, the only changes involved (a) changing all calls to opal_event functions to reflect the new framework instead, and (b) ensuring that all opal_event_t objects are properly constructed since they are now true opal_objects.
Note: Shiqing has just returned from vacation and has not yet had a chance to complete the Windows integration. Thus, this commit almost certainly breaks Windows support on the trunk. However, I want this to have a chance to soak for as long as possible before I become less available a week from today (going to be at a class for 5 days, and thus will only be sparingly available) so we can find and fix any problems.
Biggest change is moving the libevent code from opal/event to a new opal/mca/event framework. This was done to make it much easier to update libevent in the future. New versions can be inserted as a new component and tested in parallel with the current version until validated, then we can remove the earlier version if we so choose. This is a statically built framework ala installdirs, so only one component will build at a time. There is no selection logic - the sole compiled component simply loads its function pointers into the opal_event struct.
I have gone thru the code base and converted all the libevent calls I could find. However, I cannot compile nor test every environment. It is therefore quite likely that errors remain in the system. Please keep an eye open for two things:
1. compile-time errors: these will be obvious as calls to the old functions (e.g., opal_evtimer_new) must be replaced by the new framework APIs (e.g., opal_event.evtimer_new)
2. run-time errors: these will likely show up as segfaults due to missing constructors on opal_event_t objects. It appears that it became a typical practice for people to "init" an opal_event_t by simply using memset to zero it out. This will no longer work - you must either OBJ_NEW or OBJ_CONSTRUCT an opal_event_t. I tried to catch these cases, but may have missed some. Believe me, you'll know when you hit it.
There is also the issue of the new libevent "no recursion" behavior. As I described on a recent email, we will have to discuss this and figure out what, if anything, we need to do.
This commit was SVN r23925.
2010-10-24 22:35:54 +04:00
|
|
|
static opal_event_t sigusr1_handler;
|
|
|
|
|
static opal_event_t sigusr2_handler;
|
|
|
|
|
static opal_event_t sigtstp_handler;
|
|
|
|
|
static opal_event_t sigcont_handler;
|
2010-07-18 01:03:27 +04:00
|
|
|
|
2012-08-03 01:15:35 +04:00
|
|
|
static void abort_signal_callback(int signal);
|
|
|
|
|
static void clean_abort(int fd, short flags, void *arg);
|
2010-07-18 01:03:27 +04:00
|
|
|
static void epipe_signal_callback(int fd, short flags, void *arg);
|
|
|
|
|
static void signal_forward_callback(int fd, short event, void *arg);
|
|
|
|
|
|
2012-04-06 18:23:13 +04:00
|
|
|
static void setup_sighandler(int signal, opal_event_t *ev,
|
|
|
|
|
opal_event_cbfunc_t cbfunc)
|
|
|
|
|
{
|
|
|
|
|
opal_event_signal_set(orte_event_base, ev, signal, cbfunc, ev);
|
|
|
|
|
opal_event_set_priority(ev, ORTE_ERROR_PRI);
|
|
|
|
|
opal_event_signal_add(ev, NULL);
|
|
|
|
|
}
|
|
|
|
|
|
2009-05-04 15:07:40 +04:00
|
|
|
static int rte_init(void)
|
2008-02-28 04:57:57 +03:00
|
|
|
{
|
|
|
|
|
int ret;
|
|
|
|
|
char *error = NULL;
|
2008-08-12 03:48:39 +04:00
|
|
|
char *contact_path, *jobfam_dir;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_job_t *jdata;
|
|
|
|
|
orte_node_t *node;
|
|
|
|
|
orte_proc_t *proc;
|
2011-03-13 01:50:53 +03:00
|
|
|
orte_app_context_t *app;
|
2010-01-26 01:25:13 +03:00
|
|
|
|
2012-04-06 18:23:13 +04:00
|
|
|
|
2008-06-18 07:15:56 +04:00
|
|
|
/* run the prolog */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_ess_base_std_prolog())) {
|
|
|
|
|
error = "orte_ess_base_std_prolog";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2012-04-06 18:23:13 +04:00
|
|
|
|
2010-07-18 01:03:27 +04:00
|
|
|
/* setup callback for SIGPIPE */
|
2012-04-06 18:23:13 +04:00
|
|
|
setup_sighandler(SIGPIPE, &epipe_handler, epipe_signal_callback);
|
2010-07-18 01:03:27 +04:00
|
|
|
/** setup callbacks for abort signals - from this point
|
|
|
|
|
* forward, we need to abort in a manner that allows us
|
2012-08-03 01:15:35 +04:00
|
|
|
* to cleanup. However, we cannot directly use libevent
|
|
|
|
|
* to trap these signals as otherwise we cannot respond
|
|
|
|
|
* to them if we are stuck in an event! So instead use
|
|
|
|
|
* the basic POSIX trap functions to handle the signal,
|
|
|
|
|
* and then let that signal handler do some magic to
|
|
|
|
|
* avoid the hang
|
2013-05-05 21:39:45 +04:00
|
|
|
*
|
|
|
|
|
* NOTE: posix traps don't allow us to do anything major
|
|
|
|
|
* in them, so use a pipe tied to a libevent event to
|
|
|
|
|
* reach a "safe" place where the termination event can
|
|
|
|
|
* be created
|
2010-07-18 01:03:27 +04:00
|
|
|
*/
|
2012-08-03 01:15:35 +04:00
|
|
|
pipe(term_pipe);
|
|
|
|
|
/* setup an event to attempt normal termination on signal */
|
|
|
|
|
opal_event_set(orte_event_base, &term_handler, term_pipe[0], OPAL_EV_READ, clean_abort, NULL);
|
|
|
|
|
opal_event_set_priority(&term_handler, ORTE_ERROR_PRI);
|
|
|
|
|
opal_event_add(&term_handler, NULL);
|
|
|
|
|
/* point the signal trap to a function that will activate that event */
|
|
|
|
|
signal(SIGTERM, abort_signal_callback);
|
|
|
|
|
signal(SIGINT, abort_signal_callback);
|
|
|
|
|
signal(SIGHUP, abort_signal_callback);
|
2010-07-18 01:03:27 +04:00
|
|
|
|
|
|
|
|
/** setup callbacks for signals we should foward */
|
2012-04-06 18:23:13 +04:00
|
|
|
setup_sighandler(SIGUSR1, &sigusr1_handler, signal_forward_callback);
|
|
|
|
|
setup_sighandler(SIGUSR2, &sigusr2_handler, signal_forward_callback);
|
|
|
|
|
setup_sighandler(SIGTSTP, &sigtstp_handler, signal_forward_callback);
|
|
|
|
|
setup_sighandler(SIGCONT, &sigcont_handler, signal_forward_callback);
|
2010-07-18 01:03:27 +04:00
|
|
|
signals_set = true;
|
2012-04-06 18:23:13 +04:00
|
|
|
|
2011-09-11 23:02:24 +04:00
|
|
|
#if OPAL_HAVE_HWLOC
|
|
|
|
|
{
|
|
|
|
|
hwloc_obj_t obj;
|
2011-09-12 20:21:51 +04:00
|
|
|
unsigned i, j;
|
2011-09-11 23:02:24 +04:00
|
|
|
|
|
|
|
|
/* get the local topology */
|
|
|
|
|
if (NULL == opal_hwloc_topology) {
|
2011-10-29 19:07:36 +04:00
|
|
|
if (OPAL_SUCCESS != opal_hwloc_base_get_topology()) {
|
2011-09-11 23:02:24 +04:00
|
|
|
error = "topology discovery";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2009-09-22 19:41:03 +04:00
|
|
|
}
|
2011-09-11 23:02:24 +04:00
|
|
|
|
|
|
|
|
/* remove the hostname from the topology. Unfortunately, hwloc
|
|
|
|
|
* decided to add the source hostname to the "topology", thus
|
|
|
|
|
* rendering it unusable as a pure topological description. So
|
|
|
|
|
* we remove that information here.
|
|
|
|
|
*/
|
|
|
|
|
obj = hwloc_get_root_obj(opal_hwloc_topology);
|
|
|
|
|
for (i=0; i < obj->infos_count; i++) {
|
|
|
|
|
if (NULL == obj->infos[i].name ||
|
|
|
|
|
NULL == obj->infos[i].value) {
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
if (0 == strncmp(obj->infos[i].name, "HostName", strlen("HostName"))) {
|
|
|
|
|
free(obj->infos[i].name);
|
|
|
|
|
free(obj->infos[i].value);
|
|
|
|
|
/* left justify the array */
|
|
|
|
|
for (j=i; j < obj->infos_count-1; j++) {
|
|
|
|
|
obj->infos[j] = obj->infos[j+1];
|
|
|
|
|
}
|
|
|
|
|
obj->infos[obj->infos_count-1].name = NULL;
|
|
|
|
|
obj->infos[obj->infos_count-1].value = NULL;
|
|
|
|
|
obj->infos_count--;
|
|
|
|
|
break;
|
2009-09-22 19:41:03 +04:00
|
|
|
}
|
|
|
|
|
}
|
2011-09-11 23:02:24 +04:00
|
|
|
|
2013-03-28 01:14:43 +04:00
|
|
|
if (4 < opal_output_get_verbosity(orte_ess_base_framework.framework_output)) {
|
2011-09-11 23:02:24 +04:00
|
|
|
opal_output(0, "%s Topology Info:", ORTE_NAME_PRINT(ORTE_PROC_MY_NAME));
|
|
|
|
|
opal_dss.dump(0, opal_hwloc_topology, OPAL_HWLOC_TOPO);
|
|
|
|
|
}
|
2009-09-22 19:41:03 +04:00
|
|
|
}
|
2011-09-11 23:02:24 +04:00
|
|
|
#endif
|
|
|
|
|
|
2009-09-02 22:03:10 +04:00
|
|
|
/* if we are using xml for output, put an mpirun start tag */
|
|
|
|
|
if (orte_xml_output) {
|
|
|
|
|
fprintf(orte_xml_fp, "<mpirun>\n");
|
|
|
|
|
fflush(orte_xml_fp);
|
|
|
|
|
}
|
|
|
|
|
|
2012-08-29 01:20:17 +04:00
|
|
|
/* setup the global nidmap/pidmap object */
|
|
|
|
|
orte_nidmap.bytes = NULL;
|
|
|
|
|
orte_nidmap.size = 0;
|
|
|
|
|
orte_pidmap.bytes = NULL;
|
|
|
|
|
orte_pidmap.size = 0;
|
|
|
|
|
|
2008-12-22 23:23:05 +03:00
|
|
|
/* open and setup the opal_pstat framework so we can provide
|
|
|
|
|
* process stats if requested
|
|
|
|
|
*/
|
2013-03-28 01:11:47 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&opal_pstat_base_framework, 0))) {
|
2008-12-22 23:23:05 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "opal_pstat_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_pstat_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
2012-04-06 18:23:13 +04:00
|
|
|
error = "opal_pstat_base_select";
|
2008-12-22 23:23:05 +03:00
|
|
|
goto error;
|
|
|
|
|
}
|
2012-04-06 18:23:13 +04:00
|
|
|
|
|
|
|
|
/* open and setup the state machine */
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_state_base_framework, 0))) {
|
2012-04-06 18:23:13 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_state_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_state_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_state_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2013-03-28 01:14:43 +04:00
|
|
|
/* open the errmgr */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_errmgr_base_framework, 0))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
2011-08-18 20:24:45 +04:00
|
|
|
error = "orte_errmgr_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* Since we are the HNP, then responsibility for
|
|
|
|
|
* defining the name falls to the PLM component for our
|
|
|
|
|
* respective environment - hence, we have to open the PLM
|
2009-07-19 21:56:19 +04:00
|
|
|
* first and select that component.
|
2008-02-28 04:57:57 +03:00
|
|
|
*/
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_plm_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_plm_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_plm_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_plm_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-10-04 06:58:26 +04:00
|
|
|
/* if we were spawned by a singleton, our jobid was given to us */
|
|
|
|
|
if (NULL != orte_ess_base_jobid) {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_util_convert_string_to_jobid(&ORTE_PROC_MY_NAME->jobid, orte_ess_base_jobid))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "convert_string_to_jobid";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
ORTE_PROC_MY_NAME->vpid = 0;
|
|
|
|
|
} else {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_plm.set_hnp_name())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_plm_set_hnp_name";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
/* Setup the communication infrastructure */
|
2009-09-22 04:58:29 +04:00
|
|
|
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
/*
|
|
|
|
|
* OOB Layer
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_oob_base_framework, 0))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_oob_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_oob_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_oob_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/*
|
|
|
|
|
* Runtime Messaging Layer
|
|
|
|
|
*/
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_rml_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_rml_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_rml_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_rml_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2011-08-18 20:24:45 +04:00
|
|
|
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_errmgr_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_errmgr_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
/* setup the global job and node arrays */
|
|
|
|
|
orte_job_data = OBJ_NEW(opal_pointer_array_t);
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_pointer_array_init(orte_job_data,
|
2013-10-02 23:03:46 +04:00
|
|
|
1,
|
|
|
|
|
ORTE_GLOBAL_ARRAY_MAX_SIZE,
|
|
|
|
|
1))) {
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "setup job array";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
orte_node_pool = OBJ_NEW(opal_pointer_array_t);
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_pointer_array_init(orte_node_pool,
|
2013-10-02 23:03:46 +04:00
|
|
|
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
|
|
|
|
|
ORTE_GLOBAL_ARRAY_MAX_SIZE,
|
|
|
|
|
ORTE_GLOBAL_ARRAY_BLOCK_SIZE))) {
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "setup node array";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
orte_node_topologies = OBJ_NEW(opal_pointer_array_t);
|
|
|
|
|
if (ORTE_SUCCESS != (ret = opal_pointer_array_init(orte_node_topologies,
|
2013-10-02 23:03:46 +04:00
|
|
|
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
|
|
|
|
|
ORTE_GLOBAL_ARRAY_MAX_SIZE,
|
|
|
|
|
ORTE_GLOBAL_ARRAY_BLOCK_SIZE))) {
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "setup node topologies array";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* init the nidmap - just so we register that verbosity */
|
|
|
|
|
orte_util_nidmap_init(NULL);
|
|
|
|
|
|
|
|
|
|
/* Setup the job data object for the daemons */
|
|
|
|
|
/* create and store the job data object */
|
|
|
|
|
jdata = OBJ_NEW(orte_job_t);
|
|
|
|
|
jdata->jobid = ORTE_PROC_MY_NAME->jobid;
|
|
|
|
|
opal_pointer_array_set_item(orte_job_data, 0, jdata);
|
|
|
|
|
/* mark that the daemons have reported as we are the
|
|
|
|
|
* only ones in the system right now, and we definitely
|
|
|
|
|
* are running!
|
|
|
|
|
*/
|
|
|
|
|
jdata->state = ORTE_JOB_STATE_DAEMONS_REPORTED;
|
|
|
|
|
|
|
|
|
|
/* every job requires at least one app */
|
|
|
|
|
app = OBJ_NEW(orte_app_context_t);
|
|
|
|
|
opal_pointer_array_set_item(jdata->apps, 0, app);
|
|
|
|
|
jdata->num_apps++;
|
|
|
|
|
|
|
|
|
|
/* create and store a node object where we are */
|
|
|
|
|
node = OBJ_NEW(orte_node_t);
|
|
|
|
|
node->name = strdup(orte_process_info.nodename);
|
|
|
|
|
node->index = opal_pointer_array_set_item(orte_node_pool, 0, node);
|
|
|
|
|
#if OPAL_HAVE_HWLOC
|
****************************************************************
This change contains a non-mandatory modification
of the MPI-RTE interface. Anyone wishing to support
coprocessors such as the Xeon Phi may wish to add
the required definition and underlying support
****************************************************************
Add locality support for coprocessors such as the Intel Xeon Phi.
Detecting that we are on a coprocessor inside of a host node isn't straightforward. There are no good "hooks" provided for programmatically detecting that "we are on a coprocessor running its own OS", and the ORTE daemon just thinks it is on another node. However, in order to properly use the Phi's public interface for MPI transport, it is necessary that the daemon detect that it is colocated with procs on the host.
So we have to split the locality to separately record "on the same host" vs "on the same board". We already have the board-level locality flag, but not quite enough flexibility to handle this use-case. Thus, do the following:
1. add OPAL_PROC_ON_HOST flag to indicate we share a host, but not necessarily the same board
2. modify OPAL_PROC_ON_NODE to indicate we share both a host AND the same board. Note that we have to modify the OPAL_PROC_ON_LOCAL_NODE macro to explicitly check both conditions
3. add support in opal/mca/hwloc/base/hwloc_base_util.c for the host to check for coprocessors, and for daemons to check to see if they are on a coprocessor. The former is done via hwloc, but support for the latter is not yet provided by hwloc. So the code for detecting we are on a coprocessor currently is Xeon Phi specific - hopefully, we will find more generic methods in the future.
4. modify the orted and the hnp startup so they check for coprocessors and to see if they are on a coprocessor, and have the orteds pass that info back in their callback message. Automatically detect that coprocessors have been found and identify which coprocessors are on which hosts. Note that this algo isn't scalable at the moment - this will hopefully be improved over time.
5. modify the ompi proc locality detection function to look for coprocessor host info IF the OMPI_RTE_HOST_ID database key has been defined. RTE's that choose not to provide this support do not have to do anything - the associated code will simply be ignored.
6. include some cleanup of the hwloc open/close code so it conforms to how we did things in other frameworks (e.g., having a single "frame" file instead of open/close). Also, fix the locality flags - e.g., being on the same node means you must also be on the same cluster/cu, so ensure those flags are also set.
cmr:v1.7.4:reviewer=hjelmn
This commit was SVN r29435.
2013-10-14 20:52:58 +04:00
|
|
|
{
|
|
|
|
|
char *coprocessors;
|
|
|
|
|
/* add it to the array of known topologies */
|
|
|
|
|
opal_pointer_array_add(orte_node_topologies, opal_hwloc_topology);
|
|
|
|
|
/* detect and add any coprocessors */
|
|
|
|
|
coprocessors = opal_hwloc_base_find_coprocessors(opal_hwloc_topology);
|
|
|
|
|
if (NULL != coprocessors) {
|
|
|
|
|
node->coprocessors = opal_argv_split(coprocessors, ',');
|
|
|
|
|
node->coprocessor_host = true;
|
|
|
|
|
free(coprocessors);
|
|
|
|
|
orte_coprocessors_detected = true;
|
|
|
|
|
}
|
|
|
|
|
/* see if I am on a coprocessor */
|
|
|
|
|
coprocessors = opal_hwloc_base_check_on_coprocessor();
|
|
|
|
|
if (NULL != coprocessors) {
|
|
|
|
|
node->coprocessors = opal_argv_split(coprocessors, ',');
|
|
|
|
|
free(coprocessors);
|
|
|
|
|
orte_coprocessors_detected = true;
|
|
|
|
|
}
|
|
|
|
|
}
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/* create and store a proc object for us */
|
|
|
|
|
proc = OBJ_NEW(orte_proc_t);
|
|
|
|
|
proc->name.jobid = ORTE_PROC_MY_NAME->jobid;
|
|
|
|
|
proc->name.vpid = ORTE_PROC_MY_NAME->vpid;
|
|
|
|
|
|
|
|
|
|
proc->pid = orte_process_info.pid;
|
|
|
|
|
proc->rml_uri = orte_rml.get_contact_info();
|
|
|
|
|
proc->state = ORTE_PROC_STATE_RUNNING;
|
|
|
|
|
OBJ_RETAIN(node); /* keep accounting straight */
|
|
|
|
|
proc->node = node;
|
|
|
|
|
proc->nodename = node->name;
|
|
|
|
|
opal_pointer_array_set_item(jdata->procs, proc->name.vpid, proc);
|
|
|
|
|
|
|
|
|
|
/* record that the daemon (i.e., us) is on this node
|
|
|
|
|
* NOTE: we do not add the proc object to the node's
|
|
|
|
|
* proc array because we are not an application proc.
|
|
|
|
|
* Instead, we record it in the daemon field of the
|
|
|
|
|
* node object
|
|
|
|
|
*/
|
|
|
|
|
OBJ_RETAIN(proc); /* keep accounting straight */
|
|
|
|
|
node->daemon = proc;
|
|
|
|
|
node->daemon_launched = true;
|
|
|
|
|
node->state = ORTE_NODE_STATE_UP;
|
|
|
|
|
|
|
|
|
|
/* if we are to retain aliases, get ours */
|
|
|
|
|
if (orte_retain_aliases) {
|
|
|
|
|
opal_ifgetaliases(&node->alias);
|
|
|
|
|
/* add our own local name to it */
|
|
|
|
|
opal_argv_append_nosize(&node->alias, orte_process_info.nodename);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* record that the daemon job is running */
|
|
|
|
|
jdata->num_procs = 1;
|
|
|
|
|
jdata->state = ORTE_JOB_STATE_RUNNING;
|
|
|
|
|
/* obviously, we have "reported" */
|
|
|
|
|
jdata->num_reported = 1;
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/*
|
|
|
|
|
* Routed system
|
|
|
|
|
*/
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_routed_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
2013-03-28 01:14:43 +04:00
|
|
|
error = "orte_rml_base_open";
|
2008-02-28 04:57:57 +03:00
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_routed_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_routed_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2012-06-19 17:38:42 +04:00
|
|
|
/* database */
|
2013-03-28 01:11:47 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&opal_db_base_framework, 0))) {
|
2012-06-19 17:38:42 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_db_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-10-02 23:03:46 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = opal_db_base_select(true))) {
|
2012-06-19 17:38:42 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_db_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-09-27 04:37:49 +04:00
|
|
|
/* set our id */
|
|
|
|
|
opal_db.set_id((opal_identifier_t*)ORTE_PROC_MY_NAME);
|
2012-06-19 17:38:42 +04:00
|
|
|
|
2010-02-25 04:11:29 +03:00
|
|
|
/*
|
|
|
|
|
* Group communications
|
|
|
|
|
*/
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_grpcomm_base_framework, 0))) {
|
2010-02-25 04:11:29 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_grpcomm_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_grpcomm_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_grpcomm_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* Now provide a chance for the PLM
|
|
|
|
|
* to perform any module-specific init functions. This
|
|
|
|
|
* needs to occur AFTER the communications are setup
|
|
|
|
|
* as it may involve starting a non-blocking recv
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_plm.init())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_plm_init";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Setup the remaining resource
|
|
|
|
|
* management and errmgr frameworks - application procs
|
|
|
|
|
* and daemons do not open these frameworks as they only use
|
|
|
|
|
* the hnp proxy support in the PLM framework.
|
|
|
|
|
*/
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_ras_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_ras_base_open";
|
|
|
|
|
goto error;
|
2013-03-28 01:14:43 +04:00
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_ras_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_ras_base_find_available";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_rmaps_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_rmaps_base_open";
|
|
|
|
|
goto error;
|
2013-03-28 01:14:43 +04:00
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_rmaps_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_rmaps_base_find_available";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-10-08 19:02:47 +04:00
|
|
|
#if OPAL_HAVE_HWLOC
|
2013-09-21 23:47:41 +04:00
|
|
|
/* if a topology file was given, then the rmaps framework open
|
|
|
|
|
* will have reset our topology. Ensure we always get the right
|
|
|
|
|
* one by setting our node topology afterwards
|
|
|
|
|
*/
|
|
|
|
|
node->topology = opal_hwloc_topology;
|
2013-10-08 19:02:47 +04:00
|
|
|
#endif
|
2013-09-21 23:47:41 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* Open/select the odls */
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_odls_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_odls_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_odls_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_odls_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* enable communication with the rml */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_rml.enable_comm())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_rml.enable_comm";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2009-06-25 07:00:53 +04:00
|
|
|
/* we are an hnp, so update the contact info field for later use */
|
|
|
|
|
orte_process_info.my_hnp_uri = orte_rml.get_contact_info();
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
proc->rml_uri = strdup(orte_process_info.my_hnp_uri);
|
|
|
|
|
|
2009-06-25 07:00:53 +04:00
|
|
|
/* we are also officially a daemon, so better update that field too */
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
orte_process_info.my_daemon_uri = strdup(orte_process_info.my_hnp_uri);
|
2009-06-25 07:00:53 +04:00
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
/* setup the orte_show_help system to recv remote output */
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
orte_rml.recv_buffer_nb(ORTE_NAME_WILDCARD, ORTE_RML_TAG_SHOW_HELP,
|
|
|
|
|
ORTE_RML_PERSISTENT, orte_show_help_recv, NULL);
|
This commit represents a bunch of work on a Mercurial side branch. As
such, the commit message back to the master SVN repository is fairly
long.
= ORTE Job-Level Output Messages =
Add two new interfaces that should be used for all new code throughout
the ORTE and OMPI layers (we already make the search-and-replace on
the existing ORTE / OMPI layers):
* orte_output(): (and corresponding friends ORTE_OUTPUT,
orte_output_verbose, etc.) This function sends the output directly
to the HNP for processing as part of a job-specific output
channel. It supports all the same outputs as opal_output()
(syslog, file, stdout, stderr), but for stdout/stderr, the output
is sent to the HNP for processing and output. More on this below.
* orte_show_help(): This function is a drop-in-replacement for
opal_show_help(), with two differences in functionality:
1. the rendered text help message output is sent to the HNP for
display (rather than outputting directly into the process' stderr
stream)
1. the HNP detects duplicate help messages and does not display them
(so that you don't see the same error message N times, once from
each of your N MPI processes); instead, it counts "new" instances
of the help message and displays a message every ~5 seconds when
there are new ones ("I got X new copies of the help message...")
opal_show_help and opal_output still exist, but they only output in
the current process. The intent for the new orte_* functions is that
they can apply job-level intelligence to the output. As such, we
recommend that all new ORTE and OMPI code use the new orte_*
functions, not thei opal_* functions.
=== New code ===
For ORTE and OMPI programmers, here's what you need to do differently
in new code:
* Do not include opal/util/show_help.h or opal/util/output.h.
Instead, include orte/util/output.h (this one header file has
declarations for both the orte_output() series of functions and
orte_show_help()).
* Effectively s/opal_output/orte_output/gi throughout your code.
Note that orte_output_open() takes a slightly different argument
list (as a way to pass data to the filtering stream -- see below),
so you if explicitly call opal_output_open(), you'll need to
slightly adapt to the new signature of orte_output_open().
* Literally s/opal_show_help/orte_show_help/. The function signature
is identical.
=== Notes ===
* orte_output'ing to stream 0 will do similar to what
opal_output'ing did, so leaving a hard-coded "0" as the first
argument is safe.
* For systems that do not use ORTE's RML or the HNP, the effect of
orte_output_* and orte_show_help will be identical to their opal
counterparts (the additional information passed to
orte_output_open() will be lost!). Indeed, the orte_* functions
simply become trivial wrappers to their opal_* counterparts. Note
that we have not tested this; the code is simple but it is quite
possible that we mucked something up.
= Filter Framework =
Messages sent view the new orte_* functions described above and
messages output via the IOF on the HNP will now optionally be passed
through a new "filter" framework before being output to
stdout/stderr. The "filter" OPAL MCA framework is intended to allow
preprocessing to messages before they are sent to their final
destinations. The first component that was written in the filter
framework was to create an XML stream, segregating all the messages
into different XML tags, etc. This will allow 3rd party tools to read
the stdout/stderr from the HNP and be able to know exactly what each
text message is (e.g., a help message, another OMPI infrastructure
message, stdout from the user process, stderr from the user process,
etc.).
Filtering is not active by default. Filter components must be
specifically requested, such as:
{{{
$ mpirun --mca filter xml ...
}}}
There can only be one filter component active.
= New MCA Parameters =
The new functionality described above introduces two new MCA
parameters:
* '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that
help messages will be aggregated, as described above. If set to 0,
all help messages will be displayed, even if they are duplicates
(i.e., the original behavior).
* '''orte_base_show_output_recursions''': An MCA parameter to help
debug one of the known issues, described below. It is likely that
this MCA parameter will disappear before v1.3 final.
= Known Issues =
* The XML filter component is not complete. The current output from
this component is preliminary and not real XML. A bit more work
needs to be done to configure.m4 search for an appropriate XML
library/link it in/use it at run time.
* There are possible recursion loops in the orte_output() and
orte_show_help() functions -- e.g., if RML send calls orte_output()
or orte_show_help(). We have some ideas how to fix these, but
figured that it was ok to commit before feature freeze with known
issues. The code currently contains sub-optimal workarounds so
that this will not be a problem, but it would be good to actually
solve the problem rather than have hackish workarounds before v1.3 final.
This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* setup my session directory */
|
2010-03-02 18:18:33 +03:00
|
|
|
if (orte_create_session_dirs) {
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_debug_output,
|
2010-03-02 18:18:33 +03:00
|
|
|
"%s setting up session dir with\n\ttmpdir: %s\n\thost %s",
|
2009-03-06 00:50:47 +03:00
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
2010-03-02 18:18:33 +03:00
|
|
|
(NULL == orte_process_info.tmpdir_base) ? "UNDEF" : orte_process_info.tmpdir_base,
|
|
|
|
|
orte_process_info.nodename));
|
|
|
|
|
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_session_dir(true,
|
|
|
|
|
orte_process_info.tmpdir_base,
|
|
|
|
|
orte_process_info.nodename, NULL,
|
|
|
|
|
ORTE_PROC_MY_NAME))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_session_dir";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Once the session directory location has been established, set
|
2013-10-02 23:03:46 +04:00
|
|
|
the opal_output hnp file location to be in the
|
|
|
|
|
proc-specific session directory. */
|
2010-03-02 18:18:33 +03:00
|
|
|
opal_output_set_output_file_info(orte_process_info.proc_session_dir,
|
|
|
|
|
"output-", NULL, NULL);
|
|
|
|
|
|
|
|
|
|
/* save my contact info in a file for others to find */
|
|
|
|
|
jobfam_dir = opal_dirname(orte_process_info.job_session_dir);
|
|
|
|
|
contact_path = opal_os_path(false, jobfam_dir, "contact.txt", NULL);
|
|
|
|
|
free(jobfam_dir);
|
|
|
|
|
|
2008-06-09 18:53:58 +04:00
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_debug_output,
|
2010-03-02 18:18:33 +03:00
|
|
|
"%s writing contact file %s",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
contact_path));
|
|
|
|
|
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_write_hnp_contact_file(contact_path))) {
|
|
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_debug_output,
|
|
|
|
|
"%s writing contact file failed with error %s",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
|
|
|
|
ORTE_ERROR_NAME(ret)));
|
|
|
|
|
} else {
|
|
|
|
|
OPAL_OUTPUT_VERBOSE((2, orte_debug_output,
|
|
|
|
|
"%s wrote contact file",
|
|
|
|
|
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME)));
|
|
|
|
|
}
|
|
|
|
|
free(contact_path);
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* setup the routed info - the selected routed component
|
|
|
|
|
* will know what to do.
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_routed.init_routes(ORTE_PROC_MY_NAME->jobid, NULL))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_routed.init_routes";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code:
1. completely and cleanly separates responsibilities between the HNP, orted, and tool components.
2. removes all wireup messaging during launch and shutdown.
3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol.
4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0.
5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none".
6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout.
7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output"
This is not intended for the 1.3 release as it is a major change requiring considerable soak time.
This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
|
|
|
/* setup I/O forwarding system - must come after we init routes */
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_iof_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_iof_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_iof_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_iof_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* setup the FileM */
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_filem_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_filem_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_filem_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_filem_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2010-03-13 02:57:50 +03:00
|
|
|
#if OPAL_ENABLE_FT_CR == 1
|
2008-02-28 04:57:57 +03:00
|
|
|
/*
|
|
|
|
|
* Setup the SnapC
|
|
|
|
|
*/
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_snapc_base_framework, 0))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_snapc_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_sstore_base_framework, 0))) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_sstore_base_open";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2009-05-04 15:07:40 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_snapc_base_select(ORTE_PROC_IS_HNP, !ORTE_PROC_IS_DAEMON))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_snapc_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = orte_sstore_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_sstore_base_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
|
|
|
|
|
/* For HNP, ORTE doesn't need the OPAL CR stuff */
|
|
|
|
|
opal_cr_set_enabled(false);
|
|
|
|
|
#else
|
|
|
|
|
opal_cr_set_enabled(false);
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initalize the CR setup
|
|
|
|
|
* Note: Always do this, even in non-FT builds.
|
|
|
|
|
* If we don't some user level tools may hang.
|
|
|
|
|
*/
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_cr_init())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_cr_init";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2010-04-27 02:15:57 +04:00
|
|
|
/* setup the SENSOR framework */
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_sensor_base_framework, 0))) {
|
2010-04-27 02:15:57 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
2013-03-28 01:14:43 +04:00
|
|
|
error = "orte_sensor_base_open";
|
2010-04-27 02:15:57 +04:00
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_sensor_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
2010-05-05 04:48:43 +04:00
|
|
|
error = "orte_sensor_select";
|
2010-04-27 02:15:57 +04:00
|
|
|
goto error;
|
|
|
|
|
}
|
2010-05-05 04:48:43 +04:00
|
|
|
/* start the local sensors */
|
|
|
|
|
orte_sensor.start(ORTE_PROC_MY_NAME->jobid);
|
|
|
|
|
|
2012-10-25 21:15:17 +04:00
|
|
|
/* setup the dfs framework */
|
2013-03-28 01:14:43 +04:00
|
|
|
if (ORTE_SUCCESS != (ret = mca_base_framework_open(&orte_dfs_base_framework, 0))) {
|
2012-10-25 21:15:17 +04:00
|
|
|
ORTE_ERROR_LOG(ret);
|
2013-03-28 01:14:43 +04:00
|
|
|
error = "orte_dfs_base_open";
|
2012-10-25 21:15:17 +04:00
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_dfs_base_select())) {
|
|
|
|
|
ORTE_ERROR_LOG(ret);
|
|
|
|
|
error = "orte_dfs_select";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
|
2009-09-10 01:00:27 +04:00
|
|
|
/* if a tool has launched us and is requesting event reports,
|
|
|
|
|
* then set its contact info into the comm system
|
|
|
|
|
*/
|
|
|
|
|
if (orte_report_events) {
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_util_comm_connect_tool(orte_report_events_uri))) {
|
|
|
|
|
error = "could not connect to tool";
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2009-06-25 07:00:53 +04:00
|
|
|
/* We actually do *not* want an HNP to voluntarily yield() the
|
2013-10-02 23:03:46 +04:00
|
|
|
processor more than necessary. Orterun already blocks when
|
|
|
|
|
it is doing nothing, so it doesn't use any more CPU cycles than
|
|
|
|
|
it should; but when it *is* doing something, we do not want it
|
|
|
|
|
to be unnecessarily delayed because it voluntarily yielded the
|
|
|
|
|
processor in the middle of its work.
|
2009-06-25 07:00:53 +04:00
|
|
|
|
2013-10-02 23:03:46 +04:00
|
|
|
For example: when a message arrives at orterun, we want the
|
|
|
|
|
OS to wake us up in a timely fashion (which most OS's
|
|
|
|
|
seem good about doing) and then we want orterun to process
|
|
|
|
|
the message as fast as possible. If orterun yields and lets
|
|
|
|
|
aggressive MPI applications get the processor back, it may be a
|
|
|
|
|
long time before the OS schedules orterun to run again
|
|
|
|
|
(particularly if there is no IO event to wake it up). Hence,
|
|
|
|
|
routed OOB messages (for example) may be significantly delayed
|
|
|
|
|
before being delivered to MPI processes, which can be
|
|
|
|
|
problematic in some scenarios (e.g., COMM_SPAWN, BTL's that
|
|
|
|
|
require OOB messages for wireup, etc.). */
|
2009-06-25 07:00:53 +04:00
|
|
|
opal_progress_set_yield_when_idle(false);
|
2012-04-06 18:23:13 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
return ORTE_SUCCESS;
|
|
|
|
|
|
2013-10-02 23:03:46 +04:00
|
|
|
error:
|
2011-11-23 01:24:35 +04:00
|
|
|
if (ORTE_ERR_SILENT != ret && !orte_report_silent_errors) {
|
2008-06-24 21:53:10 +04:00
|
|
|
orte_show_help("help-orte-runtime.txt",
|
|
|
|
|
"orte_init:startup:internal-failure",
|
|
|
|
|
true, error, ORTE_ERROR_NAME(ret), ret);
|
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
|
2011-11-02 21:42:06 +04:00
|
|
|
return ORTE_ERR_SILENT;
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static int rte_finalize(void)
|
|
|
|
|
{
|
|
|
|
|
char *contact_path;
|
2012-08-22 21:39:22 +04:00
|
|
|
char *jobfam_dir;
|
2009-01-08 17:25:56 +03:00
|
|
|
|
2010-07-18 01:03:27 +04:00
|
|
|
if (signals_set) {
|
|
|
|
|
/* Remove the epipe handler */
|
2010-10-28 19:22:46 +04:00
|
|
|
opal_event_signal_del(&epipe_handler);
|
2012-08-12 05:28:23 +04:00
|
|
|
/* remove the term handler */
|
|
|
|
|
opal_event_del(&term_handler);
|
2010-07-18 01:03:27 +04:00
|
|
|
/** Remove the USR signal handlers */
|
2010-10-28 19:22:46 +04:00
|
|
|
opal_event_signal_del(&sigusr1_handler);
|
|
|
|
|
opal_event_signal_del(&sigusr2_handler);
|
2010-07-18 01:03:27 +04:00
|
|
|
if (orte_forward_job_control) {
|
2010-10-28 19:22:46 +04:00
|
|
|
opal_event_signal_del(&sigtstp_handler);
|
|
|
|
|
opal_event_signal_del(&sigcont_handler);
|
2010-07-18 01:03:27 +04:00
|
|
|
}
|
|
|
|
|
signals_set = false;
|
|
|
|
|
}
|
|
|
|
|
|
2010-05-05 04:48:43 +04:00
|
|
|
/* stop the local sensors */
|
|
|
|
|
orte_sensor.stop(ORTE_PROC_MY_NAME->jobid);
|
2013-03-28 01:14:43 +04:00
|
|
|
(void) mca_base_framework_close(&orte_sensor_base_framework);
|
|
|
|
|
|
|
|
|
|
/* close the dfs */
|
|
|
|
|
(void) mca_base_framework_close(&orte_dfs_base_framework);
|
|
|
|
|
(void) mca_base_framework_close(&orte_filem_base_framework);
|
|
|
|
|
/* output any lingering stdout/err data */
|
|
|
|
|
fflush(stdout);
|
|
|
|
|
fflush(stderr);
|
|
|
|
|
(void) mca_base_framework_close(&orte_iof_base_framework);
|
2013-05-05 21:39:45 +04:00
|
|
|
(void) mca_base_framework_close(&orte_odls_base_framework);
|
2013-08-14 06:03:00 +04:00
|
|
|
(void) mca_base_framework_close(&orte_rmaps_base_framework);
|
|
|
|
|
(void) mca_base_framework_close(&orte_ras_base_framework);
|
|
|
|
|
(void) mca_base_framework_close(&orte_grpcomm_base_framework);
|
|
|
|
|
(void) mca_base_framework_close(&opal_db_base_framework);
|
2013-03-28 01:14:43 +04:00
|
|
|
(void) mca_base_framework_close(&orte_routed_base_framework);
|
2013-08-14 06:03:00 +04:00
|
|
|
(void) mca_base_framework_close(&orte_plm_base_framework);
|
|
|
|
|
(void) mca_base_framework_close(&orte_errmgr_base_framework);
|
2013-03-28 01:14:43 +04:00
|
|
|
(void) mca_base_framework_close(&orte_state_base_framework);
|
2010-05-05 04:48:43 +04:00
|
|
|
|
2013-08-14 06:03:00 +04:00
|
|
|
/* cleanup the pstat stuff */
|
|
|
|
|
(void) mca_base_framework_close(&opal_pstat_base_framework);
|
|
|
|
|
|
2012-11-10 18:03:07 +04:00
|
|
|
/* remove my contact info file, if we have session directories */
|
|
|
|
|
if (NULL != orte_process_info.job_session_dir) {
|
|
|
|
|
jobfam_dir = opal_dirname(orte_process_info.job_session_dir);
|
|
|
|
|
contact_path = opal_os_path(false, jobfam_dir, "contact.txt", NULL);
|
|
|
|
|
free(jobfam_dir);
|
|
|
|
|
unlink(contact_path);
|
|
|
|
|
free(contact_path);
|
|
|
|
|
}
|
2009-03-03 19:39:13 +03:00
|
|
|
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
/* shutdown the messaging frameworks */
|
|
|
|
|
(void) mca_base_framework_close(&orte_rml_base_framework);
|
|
|
|
|
(void) mca_base_framework_close(&orte_oob_base_framework);
|
|
|
|
|
|
2012-08-22 21:39:22 +04:00
|
|
|
/* ensure we scrub the session directory tree */
|
|
|
|
|
orte_session_dir_cleanup(ORTE_JOBID_WILDCARD);
|
2008-02-28 04:57:57 +03:00
|
|
|
|
2009-09-02 22:03:10 +04:00
|
|
|
/* close the xml output file, if open */
|
|
|
|
|
if (orte_xml_output) {
|
|
|
|
|
fprintf(orte_xml_fp, "</mpirun>\n");
|
|
|
|
|
fflush(orte_xml_fp);
|
|
|
|
|
if (stdout != orte_xml_fp) {
|
|
|
|
|
fclose(orte_xml_fp);
|
|
|
|
|
}
|
|
|
|
|
}
|
2013-10-02 23:03:46 +04:00
|
|
|
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
return ORTE_SUCCESS;
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void rte_abort(int status, bool report)
|
|
|
|
|
{
|
|
|
|
|
/* do NOT do a normal finalize as this will very likely
|
|
|
|
|
* hang the process. We are aborting due to an abnormal condition
|
|
|
|
|
* that precludes normal cleanup
|
|
|
|
|
*
|
|
|
|
|
* We do need to do the following bits to make sure we leave a
|
|
|
|
|
* clean environment. Taken from orte_finalize():
|
|
|
|
|
* - Assume errmgr cleans up child processes before we exit.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/* CRS cleanup since it may have a named pipe and thread active */
|
|
|
|
|
orte_cr_finalize();
|
|
|
|
|
|
2012-08-22 21:39:22 +04:00
|
|
|
/* ensure we scrub the session directory tree */
|
|
|
|
|
orte_session_dir_cleanup(ORTE_JOBID_WILDCARD);
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* - Clean out the global structures
|
|
|
|
|
* (not really necessary, but good practice)
|
|
|
|
|
*/
|
|
|
|
|
orte_proc_info_finalize();
|
|
|
|
|
|
2010-05-12 22:11:58 +04:00
|
|
|
/* just exit */
|
2008-06-26 20:10:03 +04:00
|
|
|
exit(status);
|
2008-02-28 04:57:57 +03:00
|
|
|
}
|
|
|
|
|
|
2012-08-03 01:15:35 +04:00
|
|
|
static void clean_abort(int fd, short flags, void *arg)
|
2010-07-18 01:03:27 +04:00
|
|
|
{
|
|
|
|
|
/* if we have already ordered this once, don't keep
|
|
|
|
|
* doing it to avoid race conditions
|
|
|
|
|
*/
|
2011-10-11 22:43:45 +04:00
|
|
|
if (opal_atomic_trylock(&orte_abort_inprogress_lock)) { /* returns 1 if already locked */
|
2010-07-18 01:03:27 +04:00
|
|
|
if (forcibly_die) {
|
|
|
|
|
/* kill any local procs */
|
|
|
|
|
orte_odls.kill_local_procs(NULL);
|
|
|
|
|
|
|
|
|
|
/* whack any lingering session directory files from our jobs */
|
|
|
|
|
orte_session_dir_cleanup(ORTE_JOBID_WILDCARD);
|
|
|
|
|
|
|
|
|
|
/* cleanup our data server */
|
|
|
|
|
orte_data_server_finalize();
|
|
|
|
|
|
|
|
|
|
/* exit with a non-zero status */
|
|
|
|
|
exit(ORTE_ERROR_DEFAULT_EXIT_CODE);
|
|
|
|
|
}
|
|
|
|
|
fprintf(stderr, "%s: abort is already in progress...hit ctrl-c again to forcibly terminate\n\n", orte_basename);
|
|
|
|
|
forcibly_die = true;
|
2012-08-03 05:26:18 +04:00
|
|
|
/* reset the event */
|
|
|
|
|
opal_event_add(&term_handler, NULL);
|
2010-07-18 01:03:27 +04:00
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* ensure that the forwarding of stdin stops */
|
|
|
|
|
orte_job_term_ordered = true;
|
|
|
|
|
|
|
|
|
|
/* tell us to be quiet - hey, the user killed us with a ctrl-c,
|
|
|
|
|
* so need to tell them that!
|
|
|
|
|
*/
|
|
|
|
|
orte_execute_quiet = true;
|
|
|
|
|
|
2012-04-06 18:23:13 +04:00
|
|
|
if (!orte_never_launched) {
|
|
|
|
|
/* cleanup our data server */
|
|
|
|
|
orte_data_server_finalize();
|
|
|
|
|
}
|
|
|
|
|
|
2010-07-18 01:03:27 +04:00
|
|
|
/* We are in an event handler; the job completed procedure
|
|
|
|
|
will delete the signal handler that is currently running
|
|
|
|
|
(which is a Bad Thing), so we can't call it directly.
|
|
|
|
|
Instead, we have to exit this handler and setup to call
|
|
|
|
|
job_completed() after this. */
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
ORTE_FORCED_TERMINATE(ORTE_ERROR_DEFAULT_EXIT_CODE);
|
2010-07-18 01:03:27 +04:00
|
|
|
}
|
|
|
|
|
|
2012-08-03 01:15:35 +04:00
|
|
|
static struct timeval current, last={0,0};
|
|
|
|
|
static bool first = true;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Attempt to terminate the job and wait for callback indicating
|
|
|
|
|
* the job has been aborted.
|
|
|
|
|
*/
|
|
|
|
|
static void abort_signal_callback(int fd)
|
|
|
|
|
{
|
|
|
|
|
uint8_t foo = 1;
|
2012-08-03 05:26:18 +04:00
|
|
|
char *msg = "Abort is in progress...hit ctrl-c again within 5 seconds to forcibly terminate\n\n";
|
2012-08-03 01:15:35 +04:00
|
|
|
|
|
|
|
|
/* if this is the first time thru, just get
|
|
|
|
|
* the current time
|
|
|
|
|
*/
|
|
|
|
|
if (first) {
|
|
|
|
|
first = false;
|
|
|
|
|
gettimeofday(¤t, NULL);
|
|
|
|
|
} else {
|
|
|
|
|
/* get the current time */
|
|
|
|
|
gettimeofday(¤t, NULL);
|
|
|
|
|
/* if this is within 5 seconds of the
|
|
|
|
|
* last time we were called, then just
|
|
|
|
|
* exit - we are probably stuck
|
|
|
|
|
*/
|
|
|
|
|
if ((current.tv_sec - last.tv_sec) < 5) {
|
|
|
|
|
exit(1);
|
|
|
|
|
}
|
2013-05-05 21:39:45 +04:00
|
|
|
write(1, (void*)msg, strlen(msg));
|
2012-08-03 01:15:35 +04:00
|
|
|
}
|
|
|
|
|
/* save the time */
|
|
|
|
|
last.tv_sec = current.tv_sec;
|
|
|
|
|
/* tell the event lib to attempt to abnormally terminate */
|
|
|
|
|
write(term_pipe[1], &foo, 1);
|
|
|
|
|
}
|
|
|
|
|
|
2010-07-18 01:03:27 +04:00
|
|
|
/**
|
|
|
|
|
* Deal with sigpipe errors
|
|
|
|
|
*/
|
2010-11-25 18:25:24 +03:00
|
|
|
static int sigpipe_error_count=0;
|
2010-07-18 01:03:27 +04:00
|
|
|
static void epipe_signal_callback(int fd, short flags, void *arg)
|
|
|
|
|
{
|
2010-11-25 18:25:24 +03:00
|
|
|
sigpipe_error_count++;
|
|
|
|
|
|
|
|
|
|
if (10 < sigpipe_error_count) {
|
|
|
|
|
/* time to abort */
|
2010-11-25 18:32:48 +03:00
|
|
|
opal_output(0, "%s: SIGPIPE detected on fd %d - aborting", orte_basename, fd);
|
2012-08-03 01:15:35 +04:00
|
|
|
clean_abort(0, 0, NULL);
|
2010-11-25 18:25:24 +03:00
|
|
|
}
|
|
|
|
|
|
2010-07-18 01:03:27 +04:00
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Pass user signals to the remote application processes
|
|
|
|
|
*/
|
|
|
|
|
static void signal_forward_callback(int fd, short event, void *arg)
|
|
|
|
|
{
|
Update libevent to the 2.0 series, currently at 2.0.7rc. We will update to their final release when it becomes available. Currently known errors exist in unused portions of the libevent code. This revision passes the IBM test suite on a Linux machine and on a standalone Mac.
This is a fairly intrusive change, but outside of the moving of opal/event to opal/mca/event, the only changes involved (a) changing all calls to opal_event functions to reflect the new framework instead, and (b) ensuring that all opal_event_t objects are properly constructed since they are now true opal_objects.
Note: Shiqing has just returned from vacation and has not yet had a chance to complete the Windows integration. Thus, this commit almost certainly breaks Windows support on the trunk. However, I want this to have a chance to soak for as long as possible before I become less available a week from today (going to be at a class for 5 days, and thus will only be sparingly available) so we can find and fix any problems.
Biggest change is moving the libevent code from opal/event to a new opal/mca/event framework. This was done to make it much easier to update libevent in the future. New versions can be inserted as a new component and tested in parallel with the current version until validated, then we can remove the earlier version if we so choose. This is a statically built framework ala installdirs, so only one component will build at a time. There is no selection logic - the sole compiled component simply loads its function pointers into the opal_event struct.
I have gone thru the code base and converted all the libevent calls I could find. However, I cannot compile nor test every environment. It is therefore quite likely that errors remain in the system. Please keep an eye open for two things:
1. compile-time errors: these will be obvious as calls to the old functions (e.g., opal_evtimer_new) must be replaced by the new framework APIs (e.g., opal_event.evtimer_new)
2. run-time errors: these will likely show up as segfaults due to missing constructors on opal_event_t objects. It appears that it became a typical practice for people to "init" an opal_event_t by simply using memset to zero it out. This will no longer work - you must either OBJ_NEW or OBJ_CONSTRUCT an opal_event_t. I tried to catch these cases, but may have missed some. Believe me, you'll know when you hit it.
There is also the issue of the new libevent "no recursion" behavior. As I described on a recent email, we will have to discuss this and figure out what, if anything, we need to do.
This commit was SVN r23925.
2010-10-24 22:35:54 +04:00
|
|
|
opal_event_t *signal = (opal_event_t*)arg;
|
2010-07-18 01:03:27 +04:00
|
|
|
int signum, ret;
|
|
|
|
|
|
|
|
|
|
signum = OPAL_EVENT_SIGNAL(signal);
|
|
|
|
|
if (!orte_execute_quiet){
|
|
|
|
|
fprintf(stderr, "%s: Forwarding signal %d to job\n",
|
|
|
|
|
orte_basename, signum);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/** send the signal out to the processes, including any descendants */
|
|
|
|
|
if (ORTE_SUCCESS != (ret = orte_plm.signal_job(ORTE_JOBID_WILDCARD, signum))) {
|
|
|
|
|
fprintf(stderr, "Signal %d could not be sent to the job (returned %d)",
|
|
|
|
|
signum, ret);
|
|
|
|
|
}
|
|
|
|
|
}
|