2008-05-28 17:29:58 +04:00
/*
2010-07-30 22:59:34 +04:00
* Copyright ( c ) 2004 - 2010 The Trustees of Indiana University and Indiana
2008-05-28 17:29:58 +04:00
* University Research and Technology
* Corporation . All rights reserved .
* Copyright ( c ) 2004 - 2008 The University of Tennessee and The University
* of Tennessee Research Foundation . All rights
* reserved .
* Copyright ( c ) 2004 - 2005 High Performance Computing Center Stuttgart ,
* University of Stuttgart . All rights reserved .
* Copyright ( c ) 2004 - 2005 The Regents of the University of California .
* All rights reserved .
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
* Copyright ( c ) 2007 - 2011 Cisco Systems , Inc . All rights reserved .
2010-08-09 23:28:56 +04:00
* Copyright ( c ) 2009 - 2010 Oracle and / or its affiliates . All rights reserved .
2013-02-28 05:35:55 +04:00
* Copyright ( c ) 2012 - 2013 Los Alamos National Security , LLC .
2012-08-24 01:28:05 +04:00
* All rights reserved
As per the email discussion, revise the sparse handling of hostnames so that we avoid potential infinite loops while allowing large-scale users to improve their startup time:
* add a new MCA param orte_hostname_cutoff to specify the number of nodes at which we stop including hostnames. This defaults to INT_MAX => always include hostnames. If a value is given, then we will include hostnames for any allocation smaller than the given limit.
* remove ompi_proc_get_hostname. Replace all occurrences with a direct link to ompi_proc_t's proc_hostname, protected by appropriate "if NULL"
* modify the OMPI-ORTE integration component so that any call to modex_recv automatically loads the ompi_proc_t->proc_hostname field as well as returning the requested info. Thus, any process whose modex info you retrieve will automatically receive the hostname. Note that on-demand retrieval is still enabled - i.e., if we are running under direct launch with PMI, the hostname will be fetched upon first call to modex_recv, and then the ompi_proc_t->proc_hostname field will be loaded
* removed a stale MCA param "mpi_keep_peer_hostnames" that was no longer used anywhere in the code base
* added an envar lookup in ess/pmi for the number of nodes in the allocation. Sadly, PMI itself doesn't provide that info, so we have to get it a different way. Currently, we support PBS-based systems and SLURM - for any other, rank0 will emit a warning and we assume max number of daemons so we will always retain hostnames
This commit was SVN r29052.
2013-08-20 22:59:36 +04:00
* Copyright ( c ) 2013 Intel , Inc . All rights reserved
2008-05-28 17:29:58 +04:00
* $ COPYRIGHT $
*
* Additional copyrights may follow
*
* $ HEADER $
*/
# include "orte_config.h"
# include "orte/constants.h"
# include "orte/types.h"
# ifdef HAVE_SYS_TIME_H
# include <sys/time.h>
# endif
2009-01-08 17:27:52 +03:00
# include <stdio.h>
2008-05-28 17:29:58 +04:00
2013-03-28 01:09:41 +04:00
# include "opal/mca/base/mca_base_var.h"
2012-02-01 21:40:44 +04:00
# include "opal/mca/installdirs/installdirs.h"
2009-01-08 17:27:52 +03:00
# include "opal/util/output.h"
2009-09-18 23:48:42 +04:00
# include "opal/util/argv.h"
2008-05-28 17:29:58 +04:00
# include "orte/util/proc_info.h"
2009-09-22 19:41:03 +04:00
# include "orte/mca/errmgr/errmgr.h"
2008-05-28 17:29:58 +04:00
# include "orte/runtime/runtime.h"
# include "orte/runtime/orte_globals.h"
2009-09-22 06:16:40 +04:00
static bool passed_thru = false ;
2013-03-28 01:09:41 +04:00
static int orte_progress_thread_debug_level = - 1 ;
static char * orte_timing_file = NULL ;
static char * orte_xml_file = NULL ;
static char * orte_fork_agent_string = NULL ;
static char * orte_tmpdir_base = NULL ;
static char * orte_local_tmpdir_base = NULL ;
static char * orte_remote_tmpdir_base = NULL ;
2009-09-22 06:16:40 +04:00
2008-05-28 17:29:58 +04:00
int orte_register_params ( void )
{
2013-03-28 01:09:41 +04:00
int id ;
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
opal_output_stream_t lds ;
2011-11-23 01:24:35 +04:00
2009-09-22 06:16:40 +04:00
/* only go thru this once - mpirun calls it twice, which causes
* any error messages to show up twice
*/
if ( passed_thru ) {
return ORTE_SUCCESS ;
}
passed_thru = true ;
2010-04-28 08:06:57 +04:00
/* get a clean output channel too - need to do this here because
* we use it below , and orterun and some other tools call this
* function prior to calling orte_init
*/
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
OBJ_CONSTRUCT ( & lds , opal_output_stream_t ) ;
lds . lds_want_stdout = true ;
orte_clean_output = opal_output_open ( & lds ) ;
OBJ_DESTRUCT ( & lds ) ;
2013-03-28 01:09:41 +04:00
orte_help_want_aggregate = true ;
( void ) mca_base_var_register ( " orte " , " orte " , " base " , " help_aggregate " ,
" If orte_base_help_aggregate is true, duplicate help messages will be aggregated rather than displayed individually. This can be helpful for parallel jobs that experience multiple identical failures; rather than print out the same help/failure message N times, display it once with a count of how many processes sent the same message. " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , MCA_BASE_VAR_FLAG_SETTABLE ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL_EQ ,
& orte_help_want_aggregate ) ;
2008-06-18 07:15:56 +04:00
2012-02-16 20:10:01 +04:00
/* LOOK FOR A TMP DIRECTORY BASE */
/* Several options are provided to cover a range of possibilities:
*
* ( a ) all processes need to use a specified location as the base
* for tmp directories
* ( b ) daemons on remote nodes need to use a specified location , but
* one different from that used by mpirun
* ( c ) mpirun needs to use a specified location , but one different
* from that used on remote nodes
*/
2013-03-28 01:09:41 +04:00
orte_tmpdir_base = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " tmpdir_base " ,
" Base of the session directory tree to be used by all processes " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL_EQ ,
& orte_tmpdir_base ) ;
orte_local_tmpdir_base = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " local_tmpdir_base " ,
" Base of the session directory tree to be used by orterun/mpirun " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL_EQ ,
& orte_local_tmpdir_base ) ;
orte_remote_tmpdir_base = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " remote_tmpdir_base " ,
" Base of the session directory tree on remote nodes, if required to be different from head node " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL_EQ ,
& orte_remote_tmpdir_base ) ;
2012-02-16 20:10:01 +04:00
/* if a global tmpdir was specified, then we do not allow specification
* of the local or remote values to avoid confusion
2011-07-27 23:37:17 +04:00
*/
2013-03-28 01:09:41 +04:00
if ( NULL ! = orte_tmpdir_base & &
( NULL ! = orte_local_tmpdir_base | | NULL ! = orte_remote_tmpdir_base ) ) {
2012-02-16 20:10:01 +04:00
opal_output ( orte_clean_output ,
" ------------------------------------------------------------------ \n "
" The MCA param orte_tmpdir_base was specified, which sets the base \n "
" of the temporary directory tree for all procs. However, values for \n "
" the local and/or remote tmpdir base were also given. This can lead \n "
" to confusion and is therefore not allowed. Please specify either a \n "
" global tmpdir base OR a local/remote tmpdir base value \n "
" ------------------------------------------------------------------ " ) ;
exit ( 1 ) ;
}
2013-03-28 01:09:41 +04:00
if ( NULL ! = orte_tmpdir_base ) {
2011-07-27 23:37:17 +04:00
if ( NULL ! = orte_process_info . tmpdir_base ) {
free ( orte_process_info . tmpdir_base ) ;
}
2013-03-28 01:09:41 +04:00
orte_process_info . tmpdir_base = strdup ( orte_tmpdir_base ) ;
} else if ( ORTE_PROC_IS_HNP & & NULL ! = orte_local_tmpdir_base ) {
2012-02-16 20:10:01 +04:00
/* orterun will pickup the value for its own use */
if ( NULL ! = orte_process_info . tmpdir_base ) {
free ( orte_process_info . tmpdir_base ) ;
}
2013-03-28 01:09:41 +04:00
orte_process_info . tmpdir_base = strdup ( orte_local_tmpdir_base ) ;
} else if ( ORTE_PROC_IS_DAEMON & & NULL ! = orte_remote_tmpdir_base ) {
2012-02-16 20:10:01 +04:00
/* orterun will pickup the value and forward it along, but must not
* use it in its own work . So only a daemon needs to get it , and the
* daemon will pass it down to its application procs . Note that orterun
* will pass - its - value to any procs local to it
*/
if ( NULL ! = orte_process_info . tmpdir_base ) {
free ( orte_process_info . tmpdir_base ) ;
}
2013-03-28 01:09:41 +04:00
orte_process_info . tmpdir_base = strdup ( orte_remote_tmpdir_base ) ;
2011-07-27 23:37:17 +04:00
}
2013-03-28 01:09:41 +04:00
orte_prohibited_session_dirs = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " no_session_dirs " ,
" Prohibited locations for session directories (multiple locations separated by ',', default=NULL) " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_prohibited_session_dirs ) ;
orte_create_session_dirs = true ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " create_session_dirs " ,
" Create session directories " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_create_session_dirs ) ;
2008-05-28 17:29:58 +04:00
2013-03-28 01:09:41 +04:00
orte_execute_quiet = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " execute_quiet " ,
" Do not output error and help messages " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_execute_quiet ) ;
orte_report_silent_errors = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " report_silent_errors " ,
" Report all errors, including silent ones " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_report_silent_errors ) ;
orte_debug_flag = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debug " ,
" Top-level ORTE debug switch (default: false) " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_debug_flag ) ;
orte_debug_verbosity = - 1 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debug_verbose " ,
" Verbosity level for ORTE debug messages (default: 1) " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_debug_verbosity ) ;
orte_debug_daemons_file_flag = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debug_daemons_file " ,
" Whether want stdout/stderr of daemons to go to a file or not " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_debug_daemons_file_flag ) ;
2008-05-28 17:29:58 +04:00
/* If --debug-daemons-file was specified, that also implies
2010-11-25 02:28:00 +03:00
- - debug - daemons */
2008-05-28 17:29:58 +04:00
if ( orte_debug_daemons_file_flag ) {
orte_debug_daemons_flag = true ;
2013-03-28 01:09:41 +04:00
/* value can't change */
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debug_daemons " ,
" Whether to debug the ORTE daemons or not " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_CONSTANT ,
& orte_debug_daemons_flag ) ;
} else {
orte_debug_daemons_flag = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debug_daemons " ,
" Whether to debug the ORTE daemons or not " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_debug_daemons_flag ) ;
2008-05-28 17:29:58 +04:00
}
2008-08-01 02:11:46 +04:00
2013-03-28 01:09:41 +04:00
orte_progress_thread_debug_level = - 1 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " progress_thread_debug " ,
" Debug level for ORTE progress threads " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_progress_thread_debug_level ) ;
if ( 0 < = orte_progress_thread_debug_level ) {
2012-11-15 19:54:38 +04:00
orte_progress_thread_debug = opal_output_open ( NULL ) ;
2013-03-28 01:09:41 +04:00
opal_output_set_verbosity ( orte_progress_thread_debug ,
orte_progress_thread_debug_level ) ;
2012-11-15 19:54:38 +04:00
}
2008-08-14 22:59:01 +04:00
/* do we want session output left open? */
2013-03-28 01:09:41 +04:00
orte_leave_session_attached = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " leave_session_attached " ,
" Whether applications and/or daemons should leave their sessions "
" attached so that any output can be received - this allows X forwarding "
" without all the attendant debugging output " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_leave_session_attached ) ;
2008-08-14 22:59:01 +04:00
2012-04-29 04:03:28 +04:00
/* if any debug level is set, ensure we output debug level dumps */
if ( orte_debug_flag | | orte_debug_daemons_flag | | orte_leave_session_attached ) {
orte_devel_level_output = true ;
}
2012-01-11 23:44:22 +04:00
/* See comment in orte/tools/orterun/orterun.c about this MCA
2008-08-01 02:11:46 +04:00
param ( this param is internal ) */
2013-03-28 01:09:41 +04:00
orte_in_parallel_debugger = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " in_parallel_debugger " ,
" Whether the application is being debugged "
" in a parallel debugger (default: false) " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , MCA_BASE_VAR_FLAG_INTERNAL ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_in_parallel_debugger ) ;
orte_debugger_dump_proctable = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " output_debugger_proctable " ,
" Whether or not to output the debugger proctable after launch (default: false) " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_ALL ,
& orte_debugger_dump_proctable ) ;
orte_debugger_test_daemon = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debugger_test_daemon " ,
" Name of the executable to be used to simulate a debugger colaunch (relative or absolute path) " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_debugger_test_daemon ) ;
orte_debugger_test_attach = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debugger_test_attach " ,
" Test debugger colaunch after debugger attachment " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_debugger_test_daemon ) ;
orte_debugger_check_rate = 0 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " debugger_check_rate " ,
" Set rate (in secs) for auto-detect of debugger attachment (0 => do not check) " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_debugger_check_rate ) ;
orte_do_not_launch = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " do_not_launch " ,
" Perform all necessary operations to prepare to launch the application, but do not actually launch it " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_do_not_launch ) ;
orted_spin_flag = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " daemon_spin " ,
" Have any orteds spin until we can connect a debugger to them " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orted_spin_flag ) ;
orted_debug_failure = ORTE_VPID_INVALID ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " daemon_fail " ,
" Have the specified orted fail after init for debugging purposes " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orted_debug_failure ) ;
orted_debug_failure_delay = 0 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " daemon_fail_delay " ,
" Have the specified orted fail after specified number of seconds (default: 0 => no delay) " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orted_debug_failure_delay ) ;
orte_startup_timeout = 0 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " startup_timeout " ,
2013-11-12 03:50:40 +04:00
" Seconds to wait for startup or job launch before declaring failed_to_start (default: 0 => do not check) " ,
2013-03-28 01:09:41 +04:00
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_startup_timeout ) ;
2008-06-03 01:46:34 +04:00
2008-05-28 17:29:58 +04:00
/* check for timing requests */
2013-03-28 01:09:41 +04:00
orte_timing_details = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " timing_details " ,
" Request that detailed timing data by reported " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_timing_details ) ;
/* ensure the timing flag is set too */
orte_timing = orte_timing_details ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " timing " ,
" Request that critical timing loops be measured " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_timing ) ;
2008-05-28 17:29:58 +04:00
2009-05-04 15:07:40 +04:00
if ( ORTE_PROC_IS_HNP ) {
2013-03-28 01:09:41 +04:00
orte_timing_file = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " timing_file " ,
" Name of the file where timing data is to be written (relative or absolute path) " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_timing_file ) ;
if ( orte_timing & & NULL = = orte_timing_file ) {
2009-01-08 17:27:52 +03:00
/* send the timing output to stdout */
orte_timing_output = stdout ;
2013-03-28 01:09:41 +04:00
} else if ( NULL ! = orte_timing_file ) {
2009-01-08 17:27:52 +03:00
/* make sure the timing flag is set */
orte_timing = true ;
/* send the output to the indicated file */
2013-03-28 01:09:41 +04:00
orte_timing_output = fopen ( orte_timing_file , " w " ) ;
2009-01-08 17:27:52 +03:00
if ( NULL = = orte_timing_output ) {
/* couldn't be opened */
2013-03-28 01:09:41 +04:00
opal_output ( 0 , " File %s could not be opened " , orte_timing_file ) ;
2009-01-08 17:27:52 +03:00
orte_timing_output = stderr ;
}
}
}
2008-05-28 17:29:58 +04:00
/* User-level debugger info string */
2013-07-25 19:42:01 +04:00
orte_base_user_debugger = " totalview @mpirun@ -a @mpirun_args@ : ddt -n @np@ -start @executable@ @executable_argv@ @single_app@ : fxp @mpirun@ -a @mpirun_args@ " ;
2013-03-28 01:09:41 +04:00
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " base_user_debugger " ,
" Sequence of user-level debuggers to search for in orterun " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_base_user_debugger ) ;
#if 0
2008-05-28 17:29:58 +04:00
mca_base_param_reg_int_name ( " orte " , " abort_timeout " ,
" Max time to wait [in secs] before aborting an ORTE operation (default: 1sec) " ,
false , false , 1 , & value ) ;
orte_max_timeout = 1000000.0 * value ; /* convert to usec */
2013-03-28 01:09:41 +04:00
2008-05-28 17:29:58 +04:00
mca_base_param_reg_int_name ( " orte " , " timeout_step " ,
2008-06-03 01:46:34 +04:00
" Time to wait [in usecs/proc] before aborting an ORTE operation (default: 1000 usec/proc) " ,
false , false , 1000 , & orte_timeout_usec_per_proc ) ;
2013-03-28 01:09:41 +04:00
# endif
2008-05-28 17:29:58 +04:00
/* default hostfile */
2013-03-28 01:09:41 +04:00
orte_default_hostfile = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " default_hostfile " ,
" Name of the default hostfile (relative or absolute path, \" none \" to ignore environmental or default MCA param setting) " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_default_hostfile ) ;
if ( NULL = = orte_default_hostfile ) {
2012-02-15 08:16:05 +04:00
/* nothing was given, so define the default */
2012-05-15 20:13:39 +04:00
asprintf ( & orte_default_hostfile , " %s/openmpi-default-hostfile " , opal_install_dirs . sysconfdir ) ;
2012-02-15 08:16:05 +04:00
/* flag that nothing was given */
orte_default_hostfile_given = false ;
2013-03-28 01:09:41 +04:00
} else if ( 0 = = strcmp ( orte_default_hostfile , " none " ) ) {
free ( orte_default_hostfile ) ;
2012-02-01 21:40:44 +04:00
orte_default_hostfile = NULL ;
2012-02-15 08:16:05 +04:00
/* flag that it was given */
orte_default_hostfile_given = true ;
} else {
/* flag that it was given */
orte_default_hostfile_given = true ;
2012-02-01 21:40:44 +04:00
}
2008-05-28 17:29:58 +04:00
2011-07-07 22:54:30 +04:00
/* regex of nodes in system */
2013-03-28 01:09:41 +04:00
orte_node_regex = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " node_regex " ,
" Regular expression defining nodes in the system " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_node_regex ) ;
2011-07-07 22:54:30 +04:00
2008-05-28 17:29:58 +04:00
/* whether or not to keep FQDN hostnames */
2013-03-28 01:09:41 +04:00
orte_keep_fqdn_hostnames = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " keep_fqdn_hostnames " ,
" Whether or not to keep FQDN hostnames [default: no] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_keep_fqdn_hostnames ) ;
2008-05-28 17:29:58 +04:00
2012-11-16 08:04:29 +04:00
/* whether or not to retain aliases of hostnames */
2013-03-28 01:09:41 +04:00
orte_retain_aliases = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " retain_aliases " ,
" Whether or not to keep aliases for host names [default: no] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_retain_aliases ) ;
2012-11-16 08:04:29 +04:00
2013-01-18 09:00:05 +04:00
/* which alias to use in MPIR_proctab */
2013-03-28 01:09:41 +04:00
orte_use_hostname_alias = 1 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " hostname_alias_index " ,
" If hostname aliases are being retained, which one to use for the debugger proc table [default: 1st alias] " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_use_hostname_alias ) ;
orte_xml_output = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " xml_output " ,
" Display all output in XML format (default: false) " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_xml_output ) ;
2013-01-18 09:00:05 +04:00
2009-01-31 01:47:30 +03:00
/* whether to tag output */
2009-09-02 22:03:10 +04:00
/* if we requested xml output, be sure to tag the output as well */
2013-03-28 01:09:41 +04:00
orte_tag_output = orte_xml_output ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " tag_output " ,
" Tag all output with [job,rank] (default: false) " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_tag_output ) ;
2009-09-02 22:03:10 +04:00
if ( orte_xml_output ) {
orte_tag_output = true ;
}
2013-03-28 01:09:41 +04:00
orte_xml_file = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " xml_file " ,
" Provide all output in XML format to the specified file " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_xml_file ) ;
if ( NULL ! = orte_xml_file ) {
2009-09-02 22:03:10 +04:00
if ( ORTE_PROC_IS_HNP & & NULL = = orte_xml_fp ) {
/* only the HNP opens this file! Make sure it only happens once */
2013-03-28 01:09:41 +04:00
orte_xml_fp = fopen ( orte_xml_file , " w " ) ;
2009-09-02 22:03:10 +04:00
if ( NULL = = orte_xml_fp ) {
2013-03-28 01:09:41 +04:00
opal_output ( 0 , " Could not open specified xml output file: %s " , orte_xml_file ) ;
2009-09-02 22:03:10 +04:00
return ORTE_ERROR ;
}
}
/* ensure we set the flags to tag output */
orte_xml_output = true ;
2009-01-20 19:58:31 +03:00
orte_tag_output = true ;
2009-09-02 22:03:10 +04:00
} else {
/* default to stdout */
orte_xml_fp = stdout ;
2009-01-20 19:58:31 +03:00
}
2009-09-02 22:03:10 +04:00
2009-01-31 01:47:30 +03:00
/* whether to timestamp output */
2013-03-28 01:09:41 +04:00
orte_timestamp_output = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " timestamp_output " ,
" Timestamp all application process output (default: false) " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_timestamp_output ) ;
2009-01-31 01:47:30 +03:00
/* redirect output into files */
2013-03-28 01:09:41 +04:00
orte_output_filename = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " output_filename " ,
" Redirect output from application processes into filename.rank [default: NULL] " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_output_filename ) ;
orte_show_resolved_nodenames = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " show_resolved_nodenames " ,
" Display any node names that are resolved to a different name (default: false) " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_show_resolved_nodenames ) ;
#if 0
/* XXX -- option doesn't appear to do anything */
2008-06-24 21:50:56 +04:00
mca_base_param_reg_int_name ( " orte " , " hetero_apps " ,
" Indicates that multiple app_contexts are being provided that are a mix of 32/64 bit binaries (default: false) " ,
false , false , ( int ) false , & value ) ;
orte_hetero_apps = OPAL_INT_TO_BOOL ( value ) ;
2013-03-28 01:09:41 +04:00
# endif
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
# if OPAL_HAVE_HWLOC
2013-03-28 01:09:41 +04:00
orte_hetero_nodes = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " hetero_nodes " ,
" Nodes in cluster may differ in topology, so send the topology back from each node [Default = false] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_hetero_nodes ) ;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
# endif
Per the July technical meeting:
Standardize the handling of the orte launch agent option across PLMs. This has been a consistent complaint I have received - each PLM would register its own MCA param to get input on the launch agent for remote nodes (in fact, one or two didn't, but most did). This would then get handled in various and contradictory ways.
Some PLMs would accept only a one-word input. Others accepted multi-word args such as "valgrind orted", but then some would error by putting any prefix specified on the cmd line in front of the incorrect argument.
For example, while using the rsh launcher, if you specified "valgrind orted" as your launch agent and had "--prefix foo" on you cmd line, you would attempt to execute "ssh foo/valgrind orted" - which obviously wouldn't work.
This was all -very- confusing to users, who had to know which PLM was being used so they could even set the right mca param in the first place! And since we don't warn about non-recognized or non-used mca params, half of the time they would wind up not doing what they thought they were telling us to do.
To solve this problem, we did the following:
1. removed all mca params from the individual plms for the launch agent
2. added a new mca param "orte_launch_agent" for this purpose. To further simplify for users, this comes with a new cmd line option "--launch-agent" that can take a multi-word string argument. The value of the param defaults to "orted".
3. added a PLM base function that processes the orte_launch_agent value and adds the contents to a provided argv array. This can subsequently be harvested at-will to handle multi-word values
4. modified the PLMs to use this new function. All the PLMs except for the rsh PLM required very minor change - just called the function and moved on. The rsh PLM required much larger changes as - because of the rsh/ssh cmd line limitations - we had to correctly prepend any provided prefix to the correct argv entry.
5. added a new opal_argv_join_range function that allows the caller to "join" argv entries between two specified indices
Please let me know of any problems. I tried to make this as clean as possible, but cannot compile all PLMs to ensure all is correct.
This commit was SVN r19097.
2008-07-30 22:26:24 +04:00
/* allow specification of the launch agent */
2013-08-09 23:50:28 +04:00
orte_launch_agent = " orted " ;
2013-03-28 01:09:41 +04:00
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " launch_agent " ,
" Command used to start processes on remote nodes (default: orted) " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_launch_agent ) ;
orte_fork_agent_string = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " fork_agent " ,
" Command used to fork processes on remote nodes (default: NULL) " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_fork_agent_string ) ;
if ( NULL ! = orte_fork_agent_string ) {
orte_fork_agent = opal_argv_split ( orte_fork_agent_string , ' ' ) ;
2011-06-30 07:12:38 +04:00
}
2008-08-04 18:25:19 +04:00
/* whether or not to require RM allocation */
2013-03-28 01:09:41 +04:00
orte_allocation_required = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " allocation_required " ,
" Whether or not an allocation by a resource manager is required [default: no] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_allocation_required ) ;
2013-07-14 20:22:13 +04:00
/* whether or not to map stddiag to stderr */
orte_map_stddiag_to_stderr = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " map_stddiag_to_stderr " ,
" Map output from opal_output to stderr of the local process [default: no] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_map_stddiag_to_stderr ) ;
2013-03-28 01:09:41 +04:00
/* generate new terminal windows to display output from specified ranks */
orte_xterm = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " xterm " ,
" Create a new xterm window and display output from the specified ranks there [default: none] " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_xterm ) ;
2011-05-02 16:39:55 +04:00
if ( NULL ! = orte_xterm ) {
/* if an xterm request is given, we have to leave any ssh
* sessions attached so the xterm window manager can get
* back to the controlling terminal
*/
orte_leave_session_attached = true ;
2012-04-27 18:39:34 +04:00
/* also want to redirect stddiag output from opal_output
* to stderr from the process so those messages show
* up in the xterm window instead of being forwarded to mpirun
*/
orte_map_stddiag_to_stderr = true ;
2011-05-02 16:39:55 +04:00
}
2009-01-31 01:47:30 +03:00
2009-01-30 21:50:10 +03:00
/* whether or not to forward SIGTSTP and SIGCONT signals */
2013-03-28 01:09:41 +04:00
orte_forward_job_control = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " forward_job_control " ,
" Forward SIGTSTP (after converting to SIGSTOP) and SIGCONT signals to the application procs [default: no] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_forward_job_control ) ;
2009-02-09 23:44:44 +03:00
2009-06-03 03:52:02 +04:00
/* whether or not to report launch progress */
2013-03-28 01:09:41 +04:00
orte_report_launch_progress = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " report_launch_progress " ,
" Output a brief periodic report on launch progress [default: no] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_report_launch_progress ) ;
2009-08-11 06:51:27 +04:00
2010-08-09 23:28:56 +04:00
/* cluster hardware info detected by orte only */
2013-03-28 01:09:41 +04:00
orte_local_cpu_type = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " cpu_type " ,
" cpu type detected in node " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , MCA_BASE_VAR_FLAG_INTERNAL ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_local_cpu_type ) ;
orte_local_cpu_model = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " cpu_model " ,
" cpu model detected in node " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , MCA_BASE_VAR_FLAG_INTERNAL ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_local_cpu_model ) ;
2010-08-09 23:28:56 +04:00
2009-09-09 09:28:45 +04:00
/* tool communication controls */
2013-03-28 01:09:41 +04:00
orte_report_events_uri = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " report_events " ,
" URI to which events are to be reported (default: NULL) " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_report_events_uri ) ;
2009-09-09 09:28:45 +04:00
if ( NULL ! = orte_report_events_uri ) {
orte_report_events = true ;
}
2010-01-14 20:59:42 +03:00
/* barrier control */
2013-03-28 01:09:41 +04:00
orte_do_not_barrier = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " do_not_barrier " ,
" Do not barrier in orte_init " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , MCA_BASE_VAR_FLAG_INTERNAL ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_do_not_barrier ) ;
orte_enable_recovery = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " enable_recovery " ,
" Enable recovery from process failure [Default = disabled] " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_enable_recovery ) ;
orte_max_restarts = 0 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " max_restarts " ,
" Max number of times to restart a failed process " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_max_restarts ) ;
2011-03-20 04:17:00 +03:00
if ( ! orte_enable_recovery & & orte_max_restarts ! = 0 ) {
2010-04-28 08:06:57 +04:00
if ( ORTE_PROC_IS_HNP ) {
opal_output ( orte_clean_output ,
" ------------------------------------------------------------------ \n "
2010-07-30 22:59:34 +04:00
" The MCA param orte_enable_recovery was not set to true, but \n "
2011-03-20 04:17:00 +03:00
" a value was provided for the number of restarts: \n \n "
2011-02-14 23:49:12 +03:00
" Max restarts: %d \n "
2010-04-28 08:06:57 +04:00
" We are enabling process recovery and continuing execution. To avoid \n "
2010-07-30 22:59:34 +04:00
" this warning in the future, please set the orte_enable_recovery \n "
2010-04-28 08:06:57 +04:00
" param to non-zero. \n "
" ------------------------------------------------------------------ " ,
2011-02-14 23:49:12 +03:00
orte_max_restarts ) ;
2010-04-28 08:06:57 +04:00
}
orte_enable_recovery = true ;
}
2013-03-28 01:09:41 +04:00
orte_abort_non_zero_exit = true ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " abort_on_non_zero_status " ,
" Abort the job if any process returns a non-zero exit status - no restart in such cases " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_abort_non_zero_exit ) ;
orte_allowed_exit_without_sync = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " allowed_exit_without_sync " ,
" Process exiting without calling finalize will not trigger job termination " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_allowed_exit_without_sync ) ;
orte_staged_execution = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " staged_execution " ,
" Staged execution is being used " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_staged_execution ) ;
orte_report_child_jobs_separately = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " report_child_jobs_separately " ,
" Return the exit status of the primary job only " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_report_child_jobs_separately ) ;
2010-04-28 08:06:57 +04:00
2013-03-28 01:09:41 +04:00
#if 0
/* XXX -- unused parameter */
2010-05-12 22:11:58 +04:00
mca_base_param_reg_int_name ( " orte " , " child_time_to_exit " ,
" Max time a spawned child job is allowed to run after the primary job has terminated (seconds) " ,
false , false ,
INT_MAX , & value ) ;
orte_child_time_to_exit . tv_sec = value ;
orte_child_time_to_exit . tv_usec = 0 ;
2013-03-28 01:09:41 +04:00
# endif
2012-05-27 20:48:19 +04:00
2013-03-28 01:09:41 +04:00
orte_stat_history_size = 1 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " stat_history_size " ,
" Number of stat samples to keep " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_stat_history_size ) ;
orte_forward_envars = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " forward_envars " ,
" Comma-delimited environmental variables to forward, can include value to set " ,
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_forward_envars ) ;
orte_max_vm_size = - 1 ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " max_vm_size " ,
" Maximum size of virtual machine - used to subdivide allocation " ,
MCA_BASE_VAR_TYPE_INT , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_max_vm_size ) ;
2014-03-05 06:48:15 +04:00
# if OPAL_HAVE_HWLOC
2014-03-03 20:46:37 +04:00
if ( opal_hwloc_use_hwthreads_as_cpus ) {
orte_set_slots = strdup ( " hwthreads " ) ;
} else {
orte_set_slots = strdup ( " cores " ) ;
}
2013-03-28 01:09:41 +04:00
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " set_default_slots " ,
2014-03-03 20:46:37 +04:00
" Set the number of slots on nodes that lack such info to the "
" number of specified objects [a number, \" cores \" (default), "
" \" numas \" , \" sockets \" , \" hwthreads \" (default if hwthreads_as_cpus is set), "
" or \" none \" to skip this option] " ,
2013-03-28 01:09:41 +04:00
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_set_slots ) ;
2014-03-05 06:48:15 +04:00
# else
orte_set_slots = NULL ;
# endif
If (and only if) a user requests, set the default number of slots on any node to the number of objects of the specified type. This *only* takes effect in an unmanaged environment - i.e., if an external resource manager assigns us a number of slots, then that is what we use. However, if we are using a hostfile, then the user may or may not have given us a value for the number of slots on each node.
For those nodes (and *only* those nodes) where the user does *not* specify a slot count, we will set the number of slots according to their direction: either to the number of cores, numas, sockets, or hwthreads. Otherwise, the slot count is set to 1.
Note that the default behavior remains unchanged: in the absence of any value for #slots, and in the absence of any directive to set #slots, we will set #slots=1.
This commit was SVN r27236.
2012-09-05 00:58:26 +04:00
/* should we display the allocation after determining it? */
2013-03-28 01:09:41 +04:00
orte_display_allocation = false ;
id = mca_base_var_register ( " orte " , " orte " , NULL , " display_alloc " ,
" Whether to display the allocation after it is determined " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_display_allocation ) ;
/* register a synonym for old name -- should we remove this now? */
mca_base_var_register_synonym ( id , " orte " , " ras " , " base " , " display_alloc " , MCA_BASE_VAR_SYN_FLAG_DEPRECATED ) ;
If (and only if) a user requests, set the default number of slots on any node to the number of objects of the specified type. This *only* takes effect in an unmanaged environment - i.e., if an external resource manager assigns us a number of slots, then that is what we use. However, if we are using a hostfile, then the user may or may not have given us a value for the number of slots on each node.
For those nodes (and *only* those nodes) where the user does *not* specify a slot count, we will set the number of slots according to their direction: either to the number of cores, numas, sockets, or hwthreads. Otherwise, the slot count is set to 1.
Note that the default behavior remains unchanged: in the absence of any value for #slots, and in the absence of any directive to set #slots, we will set #slots=1.
This commit was SVN r27236.
2012-09-05 00:58:26 +04:00
/* should we display a detailed (developer-quality) version of the allocation after determining it? */
2013-03-28 01:09:41 +04:00
orte_devel_level_output = false ;
id = mca_base_var_register ( " orte " , " orte " , NULL , " display_devel_alloc " ,
" Whether to display a developer-detail allocation after it is determined " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_devel_level_output ) ;
/* register a synonym for old name -- should we remove this now? */
mca_base_var_register_synonym ( id , " orte " , " ras " , " base " , " display_devel_alloc " , MCA_BASE_VAR_SYN_FLAG_DEPRECATED ) ;
if ( orte_devel_level_output ) {
If (and only if) a user requests, set the default number of slots on any node to the number of objects of the specified type. This *only* takes effect in an unmanaged environment - i.e., if an external resource manager assigns us a number of slots, then that is what we use. However, if we are using a hostfile, then the user may or may not have given us a value for the number of slots on each node.
For those nodes (and *only* those nodes) where the user does *not* specify a slot count, we will set the number of slots according to their direction: either to the number of cores, numas, sockets, or hwthreads. Otherwise, the slot count is set to 1.
Note that the default behavior remains unchanged: in the absence of any value for #slots, and in the absence of any directive to set #slots, we will set #slots=1.
This commit was SVN r27236.
2012-09-05 00:58:26 +04:00
orte_display_allocation = true ;
}
2012-09-05 22:42:09 +04:00
/* should we treat any -host directives as "soft" - i.e., desired
* but not required
*/
2013-03-28 01:09:41 +04:00
orte_soft_locations = false ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " soft_locations " ,
" Treat -host directives as desired, but not required " ,
MCA_BASE_VAR_TYPE_BOOL , NULL , 0 , 0 ,
OPAL_INFO_LVL_9 , MCA_BASE_VAR_SCOPE_READONLY ,
& orte_soft_locations ) ;
As per the RFC, bring in the ORTE async progress code and the rewrite of OOB:
*** THIS RFC INCLUDES A MINOR CHANGE TO THE MPI-RTE INTERFACE ***
Note: during the course of this work, it was necessary to completely separate the MPI and RTE progress engines. There were multiple places in the MPI layer where ORTE_WAIT_FOR_COMPLETION was being used. A new OMPI_WAIT_FOR_COMPLETION macro was created (defined in ompi/mca/rte/rte.h) that simply cycles across opal_progress until the provided flag becomes false. Places where the MPI layer blocked waiting for RTE to complete an event have been modified to use this macro.
***************************************************************************************
I am reissuing this RFC because of the time that has passed since its original release. Since its initial release and review, I have debugged it further to ensure it fully supports tests like loop_spawn. It therefore seems ready for merge back to the trunk. Given its prior review, I have set the timeout for one week.
The code is in https://bitbucket.org/rhc/ompi-oob2
WHAT: Rewrite of ORTE OOB
WHY: Support asynchronous progress and a host of other features
WHEN: Wed, August 21
SYNOPSIS:
The current OOB has served us well, but a number of limitations have been identified over the years. Specifically:
* it is only progressed when called via opal_progress, which can lead to hangs or recursive calls into libevent (which is not supported by that code)
* we've had issues when multiple NICs are available as the code doesn't "shift" messages between transports - thus, all nodes had to be available via the same TCP interface.
* the OOB "unloads" incoming opal_buffer_t objects during the transmission, thus preventing use of OBJ_RETAIN in the code when repeatedly sending the same message to multiple recipients
* there is no failover mechanism across NICs - if the selected NIC (or its attached switch) fails, we are forced to abort
* only one transport (i.e., component) can be "active"
The revised OOB resolves these problems:
* async progress is used for all application processes, with the progress thread blocking in the event library
* each available TCP NIC is supported by its own TCP module. The ability to asynchronously progress each module independently is provided, but not enabled by default (a runtime MCA parameter turns it "on")
* multi-address TCP NICs (e.g., a NIC with both an IPv4 and IPv6 address, or with virtual interfaces) are supported - reachability is determined by comparing the contact info for a peer against all addresses within the range covered by the address/mask pairs for the NIC.
* a message that arrives on one TCP NIC is automatically shifted to whatever NIC that is connected to the next "hop" if that peer cannot be reached by the incoming NIC. If no TCP module will reach the peer, then the OOB attempts to send the message via all other available components - if none can reach the peer, then an "error" is reported back to the RML, which then calls the errmgr for instructions.
* opal_buffer_t now conforms to standard object rules re OBJ_RETAIN as we no longer "unload" the incoming object
* NIC failure is reported to the TCP component, which then tries to resend the message across any other available TCP NIC. If that doesn't work, then the message is given back to the OOB base to try using other components. If all that fails, then the error is reported to the RML, which reports to the errmgr for instructions
* obviously from the above, multiple OOB components (e.g., TCP and UD) can be active in parallel
* the matching code has been moved to the RML (and out of the OOB/TCP component) so it is independent of transport
* routing is done by the individual OOB modules (as opposed to the RML). Thus, both routed and non-routed transports can simultaneously be active
* all blocking send/recv APIs have been removed. Everything operates asynchronously.
KNOWN LIMITATIONS:
* although provision is made for component failover as described above, the code for doing so has not been fully implemented yet. At the moment, if all connections for a given peer fail, the errmgr is notified of a "lost connection", which by default results in termination of the job if it was a lifeline
* the IPv6 code is present and compiles, but is not complete. Since the current IPv6 support in the OOB doesn't work anyway, I don't consider this a blocker
* routing is performed at the individual module level, yet the active routed component is selected on a global basis. We probably should update that to reflect that different transports may need/choose to route in different ways
* obviously, not every error path has been tested nor necessarily covered
* determining abnormal termination is more challenging than in the old code as we now potentially have multiple ways of connecting to a process. Ideally, we would declare "connection failed" when *all* transports can no longer reach the process, but that requires some additional (possibly complex) code. For now, the code replicates the old behavior only somewhat modified - i.e., if a module sees its connection fail, it checks to see if it is a lifeline. If so, it notifies the errmgr that the lifeline is lost - otherwise, it notifies the errmgr that a non-lifeline connection was lost.
* reachability is determined solely on the basis of a shared subnet address/mask - more sophisticated algorithms (e.g., the one used in the tcp btl) are required to handle routing via gateways
* the RML needs to assign sequence numbers to each message on a per-peer basis. The receiving RML will then deliver messages in order, thus preventing out-of-order messaging in the case where messages travel across different transports or a message needs to be redirected/resent due to failure of a NIC
This commit was SVN r29058.
2013-08-22 20:37:40 +04:00
2014-01-31 03:50:14 +04:00
/* allow specification of the cores to be used by daemons */
orte_daemon_cores = NULL ;
( void ) mca_base_var_register ( " orte " , " orte " , NULL , " daemon_cores " ,
2014-02-15 19:11:05 +04:00
" Restrict the ORTE daemons (including mpirun) to operate on the specified cores (comma-separated list of ranges) " ,
2014-01-31 03:50:14 +04:00
MCA_BASE_VAR_TYPE_STRING , NULL , 0 , 0 ,
2014-02-15 19:11:05 +04:00
OPAL_INFO_LVL_5 , MCA_BASE_VAR_SCOPE_READONLY ,
2014-01-31 03:50:14 +04:00
& orte_daemon_cores ) ;
2008-05-28 17:29:58 +04:00
return ORTE_SUCCESS ;
}
