2013-01-28 03:25:10 +04:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2012 Los Alamos National Security, LLC. All rights reserved.
|
2013-09-10 19:34:09 +04:00
|
|
|
* Copyright (c) 2013 Mellanox Technologies, Inc.
|
|
|
|
* All rights reserved.
|
2013-01-28 03:25:10 +04:00
|
|
|
*
|
|
|
|
* $COPYRIGHT$
|
|
|
|
*
|
|
|
|
* Additional copyrights may follow
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* This is the public RTE interface to the OMPI layer. Any RTE can be
|
|
|
|
* connected to the OMPI layer by creating a new static component in
|
|
|
|
* this framework, assigning it a priority and including a configure.m4
|
|
|
|
* to define when it should be built.
|
|
|
|
*
|
|
|
|
* Each component must provide a number of types and functions that mimic
|
|
|
|
* those provided by ORTE. These include (where flexibility exists, the
|
|
|
|
* ORTE data type is shown, but any compatible type is allowed. For example,
|
|
|
|
* the jobid field in ompi_process_name_t could be any type of integer, but
|
|
|
|
* cannot be a string):
|
|
|
|
*
|
|
|
|
* (a) Process name objects and operations
|
|
|
|
* 1. Definitions for integral types ompi_jobid_t and ompi_vpid_t.
|
|
|
|
* The jobid must be unique for a given MPI_COMM_WORLD capable of
|
|
|
|
* connecting to another OMPI_COMM_WORLD and the vpid will be the
|
|
|
|
* process's rank in MPI_COMM_WORLD.
|
|
|
|
* 2. ompi_process_name_t - a struct that must contain at least two integer-typed fields:
|
|
|
|
* a. ompi_jobid_t jobid
|
|
|
|
* b. ompi_vpid_t vpid
|
|
|
|
* Note that the structure can contain any number of fields beyond these
|
|
|
|
* two, so the process name struct for any particular RTE can be whatever
|
|
|
|
* is desired.
|
|
|
|
* 3. OMPI_NAME_PRINT - a macro that prints a process name when given
|
|
|
|
* a pointer to ompi_process_name_t. The output format is to be
|
|
|
|
* a single string representing the name. This function should
|
|
|
|
* be thread-safe for multiple threads to call simultaneously.
|
|
|
|
* 4. OMPI_PROC_MY_NAME - a pointer to a global variable containing
|
|
|
|
* the ompi_process_name_t for this process. Typically, this is
|
|
|
|
* stored as a field in the ompi_process_info_t struct, but that
|
|
|
|
* is not a requirement.
|
|
|
|
* 5. OMPI_NAME_WIlDCARD - a wildcard name.
|
|
|
|
* 6. ompi_rte_compare_name_fields - a function used to compare fields
|
|
|
|
* in the ompi_process_name_t struct. The function prototype must be
|
|
|
|
* of the form:
|
|
|
|
* int ompi_rte_compare_name_fields(ompi_rte_cmp_bitmask_t mask,
|
|
|
|
* ompi_process_name_t *name1,
|
|
|
|
* ompi_process_name_t *name2);
|
|
|
|
* The bitmask must be defined to indicate the fields to be used
|
|
|
|
* in the comparison. Fields not included in the mask must be ignored.
|
|
|
|
* Supported bitmask values must include:
|
|
|
|
* b. OMPI_RTE_CMP_JOBID
|
|
|
|
* c. OMPI_RTE_CMP_VPID
|
|
|
|
* d. OMPI_RTE_CMP_ALL
|
2013-07-22 21:24:17 +04:00
|
|
|
* 7. uint64_t ompi_rte_hash_name(name) - return a string hash uniquely
|
2013-01-28 03:25:10 +04:00
|
|
|
* representing the ompi_process_name passed in.
|
|
|
|
* 8. OMPI_NAME - an Opal DSS constant for a handler already registered
|
|
|
|
* to serialize/deserialize an ompi_process_name_t structure.
|
|
|
|
*
|
|
|
|
* (b) Collective objects and operations
|
|
|
|
* 1. ompi_rte_collective_t - an OPAL object used during RTE collective operations
|
|
|
|
* such as modex and barrier. It must be an opal_list_item_t and contain the
|
|
|
|
* following fields:
|
|
|
|
* a. id (ORTE type: int32_t)
|
|
|
|
* b. bool active
|
|
|
|
* flag that user can poll on to know when collective
|
|
|
|
* has completed - set to false just prior to
|
|
|
|
* calling user callback function, if provided
|
|
|
|
* 2. ompi_rte_modex - a function that performs an exchange of endpoint information
|
|
|
|
* to wireup the MPI transports. The function prototype must be of the form:
|
|
|
|
* int ompi_rte_modex(ompi_rte_collective_t *coll);
|
|
|
|
* At the completion of the modex operation, the coll->active flag must be set
|
|
|
|
* to false, and the endpoint information must be stored in the modex database.
|
|
|
|
* This function must have barrier semantics across the MPI_COMM_WORLD of the
|
|
|
|
* calling process.
|
|
|
|
* 3. ompi_rte_barrier - a function that performs a barrier operation within the
|
|
|
|
* RTE. The function prototype must be of the form:
|
|
|
|
* int ompi_rte_barrier(ompi_rte_collective_t *coll);
|
|
|
|
* At the completion of the barrier operation, the coll->active flag must be set
|
|
|
|
* to false
|
|
|
|
*
|
|
|
|
* (c) Process info struct
|
|
|
|
* 1. ompi_process_info_t - a struct containing info about the current process.
|
|
|
|
* The struct must contain at least the following fields:
|
|
|
|
* a. app_num -
|
2013-07-21 02:14:12 +04:00
|
|
|
* b. pid - this process's pid. Should be same as getpid().
|
|
|
|
* c. num_procs - Number of processes in this job (ie, MCW)
|
|
|
|
* d. my_node_rank - relative rank on local node to other peers this run-time
|
|
|
|
* instance knows about. If doing dynamics, this may be something
|
|
|
|
* different than my_local_rank, but will be my_local_rank in a
|
|
|
|
* static job.
|
|
|
|
* d. my_local_rank - relative rank on local node with other peers in this job (ie, MCW)
|
|
|
|
* e. num_local_peers - Number of local peers (peers in MCW on your node)
|
|
|
|
* f. my_hnp_uri -
|
|
|
|
* g. peer_modex - a collective id for the modex operation
|
|
|
|
* h. peer_init_barrier - a collective id for the barrier during MPI_Init
|
|
|
|
* i. peer_fini_barrier - a collective id for the barrier during MPI_Finalize
|
|
|
|
* j. job_session_dir -
|
|
|
|
* k. proc_session_dir -
|
|
|
|
* l. nodename - a string representation for the name of the node this
|
2013-01-28 03:25:10 +04:00
|
|
|
* process is on
|
2013-07-21 02:14:12 +04:00
|
|
|
* m. cpuset -
|
|
|
|
* 2. ompi_process_info - a global instance of the ompi_process_t structure.
|
|
|
|
* 3. ompi_rte_proc_is_bound - global boolean that will be true if the runtime bound
|
2013-01-28 03:25:10 +04:00
|
|
|
* the process to a particular core or set of cores and is false otherwise.
|
|
|
|
*
|
|
|
|
* (d) Error handling objects and operations
|
|
|
|
* 1. void ompi_rte_abort(int err_code, char *fmt, ...) - Abort the current
|
|
|
|
* process with the specified error code and message.
|
|
|
|
* 2. int ompi_rte_abort_peers(ompi_process_name_t *procs, size_t nprocs) -
|
|
|
|
* Abort the specified list of peers
|
|
|
|
* 3. OMPI_ERROR_LOG(rc) - print error message regarding the given return code
|
2013-07-21 02:14:12 +04:00
|
|
|
* 4. ompi_rte_register_errhandler - register a callback function for the RTE
|
2013-01-28 03:25:10 +04:00
|
|
|
* to report asynchronous errors to the caller
|
|
|
|
*
|
|
|
|
* (e) Init and finalize objects and operations
|
|
|
|
* 1. ompi_rte_init - a function to initialize the RTE. The function
|
|
|
|
* prototype must be of the form:
|
|
|
|
* int ompi_rte_init(int *argc, char ***argv);
|
|
|
|
* 2. ompi_rte_finalize - a function to finalize the RTE. The function
|
|
|
|
* prototype must be of the form:
|
|
|
|
* int ompi_rte_finalize(void);
|
|
|
|
* 3. void ompi_rte_wait_for_debugger(void) - Called during MPI_Init, this
|
|
|
|
* function is used to wait for debuggers to do their pre-MPI attach.
|
|
|
|
* If there is no attached debugger, this function will not block.
|
|
|
|
*
|
2013-02-13 01:10:11 +04:00
|
|
|
* (f) Database operations
|
2013-01-28 03:25:10 +04:00
|
|
|
* 1. ompi_rte_db_store - a function to store modex and other data in
|
|
|
|
* a local database. The function is primarily used for storing modex
|
|
|
|
* data, but can be used for general purposes. The prototype must be
|
|
|
|
* of the form:
|
|
|
|
* int ompi_rte_db_store(const ompi_process_name_t *proc,
|
|
|
|
* const char *key, const void *data,
|
|
|
|
* opal_data_type_t type);
|
|
|
|
* The implementation of this function must store a COPY of the data
|
|
|
|
* provided - the data is NOT guaranteed to be valid after return
|
|
|
|
* from the call.
|
2013-12-19 05:16:58 +04:00
|
|
|
* 3. ompi_rte_db_fetch -
|
|
|
|
* NOTE: Fetch accepts an 'ompi_proc_t'.
|
|
|
|
* int ompi_rte_db_fetch(const struct ompi_proc_t *proc,
|
2013-01-28 03:25:10 +04:00
|
|
|
* const char *key,
|
2013-12-19 05:16:58 +04:00
|
|
|
* void **data,
|
|
|
|
* opal_data_type_t type);
|
|
|
|
* 4. ompi_rte_db_fetch_pointer -
|
|
|
|
* NOTE: Fetch accepts an 'ompi_proc_t'.
|
|
|
|
* int ompi_rte_db_fetch_pointer(const struct ompi_proc_t *proc,
|
|
|
|
* const char *key,
|
|
|
|
* void **data,
|
|
|
|
* opal_data_type_t type);
|
2013-01-28 03:25:10 +04:00
|
|
|
* 5. Pre-defined db keys (with associated values after rte_init)
|
|
|
|
* a. OMPI_DB_HOSTNAME
|
|
|
|
* b. OMPI_DB_LOCALITY
|
|
|
|
*
|
2013-02-13 01:10:11 +04:00
|
|
|
* (g) Communication support
|
2013-01-28 03:25:10 +04:00
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef OMPI_MCA_RTE_H
|
|
|
|
#define OMPI_MCA_RTE_H
|
|
|
|
|
|
|
|
#include "ompi_config.h"
|
|
|
|
|
|
|
|
#include "opal/dss/dss_types.h"
|
|
|
|
#include "opal/mca/mca.h"
|
|
|
|
#include "opal/mca/base/base.h"
|
|
|
|
|
|
|
|
BEGIN_C_DECLS
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Structure for rte components.
|
|
|
|
*/
|
|
|
|
struct ompi_rte_base_component_1_0_0_t {
|
|
|
|
/** MCA base component */
|
|
|
|
mca_base_component_t base_version;
|
|
|
|
/** MCA base data */
|
|
|
|
mca_base_component_data_t base_data;
|
|
|
|
};
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Convenience typedef
|
|
|
|
*/
|
|
|
|
typedef struct ompi_rte_base_component_1_0_0_t ompi_rte_base_component_1_0_0_t;
|
|
|
|
typedef struct ompi_rte_base_component_1_0_0_t ompi_rte_component_t;
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Macro for use in components that are of type rte
|
|
|
|
*/
|
|
|
|
#define OMPI_RTE_BASE_VERSION_1_0_0 \
|
|
|
|
MCA_BASE_VERSION_2_0_0, \
|
|
|
|
"rte", 2, 0, 0
|
|
|
|
|
|
|
|
END_C_DECLS
|
|
|
|
|
|
|
|
/* include implementation to call */
|
|
|
|
#include MCA_rte_IMPLEMENTATION_HEADER
|
|
|
|
|
|
|
|
BEGIN_C_DECLS
|
|
|
|
|
2013-09-27 04:37:49 +04:00
|
|
|
/* Each RTE is required to define a DB key for identifying the node
|
|
|
|
* upon which a process resides, and for providing this information
|
|
|
|
* for each process
|
|
|
|
*
|
|
|
|
* #define OMPI_RTE_NODE_ID
|
|
|
|
*/
|
|
|
|
|
2013-01-28 03:25:10 +04:00
|
|
|
/* Communication tags */
|
|
|
|
#define OMPI_RML_TAG_UDAPL OMPI_RML_TAG_BASE+1
|
|
|
|
#define OMPI_RML_TAG_OPENIB OMPI_RML_TAG_BASE+2
|
|
|
|
#define OMPI_RML_TAG_XOPENIB OMPI_RML_TAG_BASE+3
|
|
|
|
#define OMPI_RML_TAG_COMM_CID_INTRA OMPI_RML_TAG_BASE+4
|
|
|
|
#define OMPI_RML_TAG_XOOB OMPI_RML_TAG_BASE+5
|
|
|
|
#define OMPI_RML_TAG_SM_BACK_FILE_CREATED OMPI_RML_TAG_BASE+6
|
|
|
|
#define OMPI_CRCP_COORD_BOOKMARK_TAG OMPI_RML_TAG_BASE+7
|
|
|
|
#define OMPI_COMM_JOIN_TAG OMPI_RML_TAG_BASE+8
|
|
|
|
|
|
|
|
/* support for shared memory collectives */
|
|
|
|
#define OMPI_RML_TAG_COLL_SM2_BACK_FILE_CREATED OMPI_RML_TAG_BASE+9
|
|
|
|
/* common sm component query result index */
|
|
|
|
#define OMPI_RML_TAG_COMMON_SM_COMP_INDEX OMPI_RML_TAG_BASE+10
|
|
|
|
|
|
|
|
/* OFACM RML TAGs */
|
|
|
|
#define OMPI_RML_TAG_OFACM OMPI_RML_TAG_BASE+11
|
|
|
|
#define OMPI_RML_TAG_XOFACM OMPI_RML_TAG_BASE+12
|
|
|
|
|
2013-08-23 22:02:50 +04:00
|
|
|
#define OMPI_RML_PCONNECT_TAG OMPI_RML_TAG_BASE+13
|
|
|
|
|
2013-09-10 19:34:09 +04:00
|
|
|
/* open shmem oob communication */
|
|
|
|
#define OMPI_RML_TAG_SHMEM OMPI_RML_TAG_BASE+14
|
|
|
|
|
2013-01-28 03:25:10 +04:00
|
|
|
#define OMPI_RML_TAG_DYNAMIC OMPI_RML_TAG_BASE+200
|
|
|
|
|
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
|
|
|
/*
|
|
|
|
* MCA Framework
|
|
|
|
*/
|
|
|
|
OMPI_DECLSPEC extern mca_base_framework_t ompi_rte_base_framework;
|
|
|
|
|
|
|
|
/* In a few places, we need to barrier until something happens
|
|
|
|
* that changes a flag to indicate we can release - e.g., waiting
|
|
|
|
* for a specific RTE message to arrive. We don't want to block MPI
|
|
|
|
* progress while waiting, so we loop over opal_progress, letting
|
|
|
|
* the RTE progress thread move the RTE along
|
|
|
|
*/
|
|
|
|
#define OMPI_WAIT_FOR_COMPLETION(flg) \
|
|
|
|
do { \
|
|
|
|
opal_output_verbose(1, ompi_rte_base_framework.framework_output, \
|
|
|
|
"%s waiting on RTE event at %s:%d", \
|
|
|
|
OMPI_NAME_PRINT(OMPI_PROC_MY_NAME), \
|
|
|
|
__FILE__, __LINE__); \
|
|
|
|
while ((flg)) { \
|
|
|
|
opal_progress(); \
|
|
|
|
} \
|
|
|
|
}while(0);
|
|
|
|
|
2013-01-28 03:25:10 +04:00
|
|
|
typedef struct {
|
|
|
|
opal_list_item_t super;
|
|
|
|
ompi_process_name_t name;
|
|
|
|
} ompi_namelist_t;
|
|
|
|
OBJ_CLASS_DECLARATION(ompi_namelist_t);
|
|
|
|
|
|
|
|
END_C_DECLS
|
|
|
|
|
|
|
|
#endif /* OMPI_RTE_H_ */
|