ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
81cd3e77cf
openmpi/orte/runtime/orte_mca_params.c

563 строки
27 KiB
C
Исходник Обычный вид История

ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
/*
Some relatively minor C/R related cleanup * Fix a configure warning for checking --enable-ft-thread * In hnp and orted ErrMgr components check to see if other components have already recovered this process before trying to recover it again. * Fix 'npernode' for restarting using the resilient rmaps component * export ompi_info_set, so that internal functionality can use it. This commit was SVN r23535.
2010-07-30 22:59:34 +04:00
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
* University Research and Technology
* Corporation. All rights reserved.
* Copyright (c) 2004-2008 The University of Tennessee and The University
* of Tennessee Research Foundation. All rights
* reserved.
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
* University of Stuttgart. All rights reserved.
* Copyright (c) 2004-2005 The Regents of the University of California.
* All rights reserved.
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
* Copyright (c) 2007-2011 Cisco Systems, Inc. All rights reserved.
Added new solaris sysinfo module. Also added code to assign orte_local_chip_type and orte_local_chip_model in MPI processes it the appropriate sysinfo module found the values on the machine. This commit was SVN r23581.
2010-08-09 23:28:56 +04:00
* Copyright (c) 2009-2010 Oracle and/or its affiliates. All rights reserved.
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "orte_config.h"
#include "orte/constants.h"
#include "orte/types.h"
#ifdef HAVE_SYS_TIME_H
#include <sys/time.h>
#endif
Continue to improve timing instrumentation. Add ability to store timing data directly to a file instead of just to stdout. This commit was SVN r20229.
2009-01-08 17:27:52 +03:00
#include <stdio.h>
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
#include "opal/mca/base/mca_base_param.h"
In accordance with prior releases, we are supposed to default to looking at the openmpi-default-hostfile as a default hostfile. Restore that behavior, but ignore the file if it is empty. Allow the user to ignore any MCA param setting pointing to a default hostfile by setting the param to "none" (via cmd line or whatever) - this allows them to override a setting in the system default MCA param file. This commit was SVN r25851.
2012-02-01 21:40:44 +04:00
#include "opal/mca/installdirs/installdirs.h"
Continue to improve timing instrumentation. Add ability to store timing data directly to a file instead of just to stdout. This commit was SVN r20229.
2009-01-08 17:27:52 +03:00
#include "opal/util/output.h"
Modify the paffinity system to handle binding directives that are "soft" - i.e., when someone directs that we bind if the system supports it. This allows community members to distribute OMPI with default MCA param files that direct general binding policies, without having the distributed software fail if the system cannot support those policies. The new options work by adding an ":if-avail" qualifier to the "bind-to-socket" and "bind-to-core" MCA params. If the system does not support this capability, the job will launch anyway. Without the qualifier, the job will abort with an error message indicating that the required functionality is not supported on this system. This commit was SVN r21975.
2009-09-18 23:48:42 +04:00
#include "opal/util/argv.h"
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
#include "orte/util/proc_info.h"
Some (hopefully final!) adjustments and corrections to the paffinity support: 1. default -npersocket to force -bind-to-socket 2. if we cannot get a value for cores/socket, try using #logical cpus. otherwise, default to 1 core 3. add missing error message for not-enough-processors 4. since we no longer loop through orte_register_params twice, put the auto-detect of topology info in the rte_init for hnp and std_orted 5. fix bind-to-core, bysocket combination This commit was SVN r21992.
2009-09-22 19:41:03 +04:00
#include "orte/mca/errmgr/errmgr.h"
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
#include "orte/runtime/runtime.h"
#include "orte/runtime/orte_globals.h"
Update the cm ess module to support orted bootstrap. Continue work towards bootstrap capability. This commit was SVN r21989.
2009-09-22 06:16:40 +04:00
static bool passed_thru = false;
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
int orte_register_params(void)
{
This is actually a much smaller commit than it appears at first glance - it just touches a lot of files. The --without-rte-support configuration option has never really been implemented completely. The option caused various objects not to be defined and conditionally compiled some base functions, but did nothing to prevent build of the component libraries. Unfortunately, since many of those components use objects covered by the option, it caused builds to break if those components were allowed to build. Brian dealt with this in the past by creating platform files and using "no-build" to block the components. This was clunky, but acceptable when only one organization was using that option. However, that number has now expanded to at least two more locations. Accordingly, make --without-rte-support actually work by adding appropriate configury to prevent components from building when they shouldn't. While doing so, remove two frameworks (db and rmcast) that are no longer used as ORCM comes to a close (besides, they belonged in ORCM now anyway). Do some minor cleanups along the way. This commit was SVN r25497.
2011-11-23 01:24:35 +04:00
int value;
Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options. Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location. This commit was SVN r25941.
2012-02-16 20:10:01 +04:00
char *strval, *strval1, *strval2;
This is actually a much smaller commit than it appears at first glance - it just touches a lot of files. The --without-rte-support configuration option has never really been implemented completely. The option caused various objects not to be defined and conditionally compiled some base functions, but did nothing to prevent build of the component libraries. Unfortunately, since many of those components use objects covered by the option, it caused builds to break if those components were allowed to build. Brian dealt with this in the past by creating platform files and using "no-build" to block the components. This was clunky, but acceptable when only one organization was using that option. However, that number has now expanded to at least two more locations. Accordingly, make --without-rte-support actually work by adding appropriate configury to prevent components from building when they shouldn't. While doing so, remove two frameworks (db and rmcast) that are no longer used as ORCM comes to a close (besides, they belonged in ORCM now anyway). Do some minor cleanups along the way. This commit was SVN r25497.
2011-11-23 01:24:35 +04:00
Update the cm ess module to support orted bootstrap. Continue work towards bootstrap capability. This commit was SVN r21989.
2009-09-22 06:16:40 +04:00
/* only go thru this once - mpirun calls it twice, which causes
* any error messages to show up twice
*/
if (passed_thru) {
return ORTE_SUCCESS;
}
passed_thru = true;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
/* get a clean output channel too - need to do this here because
* we use it below, and orterun and some other tools call this
* function prior to calling orte_init
*/
{
opal_output_stream_t lds;
OBJ_CONSTRUCT(&lds, opal_output_stream_t);
lds.lds_want_stdout = true;
orte_clean_output = opal_output_open(&lds);
OBJ_DESTRUCT(&lds);
}
#endif /* !ORTE_DISABLE_FULL_SUPPORT */
Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm. Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed. This commit was SVN r18664.
2008-06-18 07:15:56 +04:00
mca_base_param_reg_int_name("orte", "base_help_aggregate",
"If orte_base_help_aggregate is true, duplicate help messages will be aggregated rather than displayed individually. This can be helpful for parallel jobs that experience multiple identical failures; rather than print out the same help/failure message N times, display it once with a count of how many processes sent the same message.",
false, false,
(int) true, &value);
orte_help_want_aggregate = OPAL_INT_TO_BOOL(value);
Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options. Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location. This commit was SVN r25941.
2012-02-16 20:10:01 +04:00
/* LOOK FOR A TMP DIRECTORY BASE */
/* Several options are provided to cover a range of possibilities:
*
* (a) all processes need to use a specified location as the base
* for tmp directories
* (b) daemons on remote nodes need to use a specified location, but
* one different from that used by mpirun
* (c) mpirun needs to use a specified location, but one different
* from that used on remote nodes
*/
Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm. Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed. This commit was SVN r18664.
2008-06-18 07:15:56 +04:00
mca_base_param_reg_string_name("orte", "tmpdir_base",
Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options. Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location. This commit was SVN r25941.
2012-02-16 20:10:01 +04:00
"Base of the session directory tree to be used by all processes",
false, false, NULL, &strval);
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options. Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location. This commit was SVN r25941.
2012-02-16 20:10:01 +04:00
mca_base_param_reg_string_name("orte", "local_tmpdir_base",
"Base of the session directory tree to be used by orterun/mpirun",
false, false, NULL, &strval1);
Add a new param "orte_remote_tmpdir_base" for those situations where the compute nodes require a different session directory head than the head node. This commit was SVN r24956.
2011-07-27 23:37:17 +04:00
mca_base_param_reg_string_name("orte", "remote_tmpdir_base",
"Base of the session directory tree on remote nodes, if required to be different from head node",
Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options. Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location. This commit was SVN r25941.
2012-02-16 20:10:01 +04:00
false, false, NULL, &strval2);
/* if a global tmpdir was specified, then we do not allow specification
* of the local or remote values to avoid confusion
Add a new param "orte_remote_tmpdir_base" for those situations where the compute nodes require a different session directory head than the head node. This commit was SVN r24956.
2011-07-27 23:37:17 +04:00
*/
Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options. Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location. This commit was SVN r25941.
2012-02-16 20:10:01 +04:00
if (NULL != strval &&
(NULL != strval1 || NULL != strval2)) {
opal_output(orte_clean_output,
"------------------------------------------------------------------\n"
"The MCA param orte_tmpdir_base was specified, which sets the base\n"
"of the temporary directory tree for all procs. However, values for\n"
"the local and/or remote tmpdir base were also given. This can lead\n"
"to confusion and is therefore not allowed. Please specify either a\n"
"global tmpdir base OR a local/remote tmpdir base value\n"
"------------------------------------------------------------------");
exit(1);
}
if (NULL != strval) {
Add a new param "orte_remote_tmpdir_base" for those situations where the compute nodes require a different session directory head than the head node. This commit was SVN r24956.
2011-07-27 23:37:17 +04:00
if (NULL != orte_process_info.tmpdir_base) {
free(orte_process_info.tmpdir_base);
}
orte_process_info.tmpdir_base = strval;
Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options. Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location. This commit was SVN r25941.
2012-02-16 20:10:01 +04:00
} else if (ORTE_PROC_IS_HNP && NULL != strval1) {
/* orterun will pickup the value for its own use */
if (NULL != orte_process_info.tmpdir_base) {
free(orte_process_info.tmpdir_base);
}
orte_process_info.tmpdir_base = strval1;
} else if (ORTE_PROC_IS_DAEMON && NULL != strval2) {
/* orterun will pickup the value and forward it along, but must not
* use it in its own work. So only a daemon needs to get it, and the
* daemon will pass it down to its application procs. Note that orterun
* will pass -its- value to any procs local to it
*/
if (NULL != orte_process_info.tmpdir_base) {
free(orte_process_info.tmpdir_base);
}
orte_process_info.tmpdir_base = strval2;
Add a new param "orte_remote_tmpdir_base" for those situations where the compute nodes require a different session directory head than the head node. This commit was SVN r24956.
2011-07-27 23:37:17 +04:00
}
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
mca_base_param_reg_string_name("orte", "no_session_dirs",
"Prohibited locations for session directories (multiple locations separated by ',', default=NULL)",
false, false, NULL, &orte_prohibited_session_dirs);
Create a new mca parameter to control creation of session directories. Defaults to true so that the current behavior of always creating them is preserved. If set to false (0), then don't create session directories. Helps in those environments where session directories are a problem. Tell the sm btl that it cannot run if no session directories were created. This commit was SVN r22756.
2010-03-02 18:18:33 +03:00
mca_base_param_reg_int_name("orte", "create_session_dirs",
"Create session directories",
Don't overwrite a prior setting for create_session_dirs This commit was SVN r23383.
2010-07-13 10:30:09 +04:00
false, false, orte_create_session_dirs, &value);
Create a new mca parameter to control creation of session directories. Defaults to true so that the current behavior of always creating them is preserved. If set to false (0), then don't create session directories. Helps in those environments where session directories are a problem. Tell the sm btl that it cannot run if no session directories were created. This commit was SVN r22756.
2010-03-02 18:18:33 +03:00
orte_create_session_dirs = OPAL_INT_TO_BOOL(value);
Truly respect the -quiet option. Make it an mca param so someone doesn't have to put it solely on the cmd line. Tell show_help to shaddup as well. This commit was SVN r22926.
2010-04-02 18:19:38 +04:00
mca_base_param_reg_int_name("orte", "execute_quiet",
"Do not output error and help messages",
false, false, (int) false, &value);
orte_execute_quiet = OPAL_INT_TO_BOOL(value);
To help with debugging, add a new mca param that instructs ORTE_ERROR_LOG to output "silent" errors. Helps to track down silent errors that don't have an associated error message (e.g., via show_help). This commit was SVN r23893.
2010-10-16 07:29:47 +04:00
mca_base_param_reg_int_name("orte", "report_silent_errors",
"Report all errors, including silent ones",
false, false, (int) false, &value);
orte_report_silent_errors = OPAL_INT_TO_BOOL(value);
Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm. Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed. This commit was SVN r18664.
2008-06-18 07:15:56 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
fixes to get compiling on Red Storm again This commit was SVN r20252.
2009-01-13 01:30:00 +03:00
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
mca_base_param_reg_int_name("orte", "debug",
"Top-level ORTE debug switch (default verbosity: 1)",
false, false, (int)false, &value);
orte_debug_flag = OPAL_INT_TO_BOOL(value);
mca_base_param_reg_int_name("orte", "debug_verbose",
"Verbosity level for ORTE debug messages (default: 1)",
false, false, -1, &orte_debug_verbosity);
Add an mca param to indicate if orte progress threads are to be enabled. Error out if this is given and libevent thread support was not built. This commit was SVN r24089.
2010-11-25 02:28:00 +03:00
mca_base_param_reg_int_name("orte", "debug_daemons",
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
"Whether to debug the ORTE daemons or not",
false, false, (int)false, &value);
orte_debug_daemons_flag = OPAL_INT_TO_BOOL(value);
mca_base_param_reg_int_name("orte", "debug_daemons_file",
"Whether want stdout/stderr of daemons to go to a file or not",
false, false, (int)false, &value);
orte_debug_daemons_file_flag = OPAL_INT_TO_BOOL(value);
/* If --debug-daemons-file was specified, that also implies
Add an mca param to indicate if orte progress threads are to be enabled. Error out if this is given and libevent thread support was not built. This commit was SVN r24089.
2010-11-25 02:28:00 +03:00
--debug-daemons */
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
if (orte_debug_daemons_file_flag) {
orte_debug_daemons_flag = true;
}
Fixes trac:1361: mainly add new internal MCA parameter that orterun will set when it launches under debuggers using the --debug option. This commit was SVN r19116. The following Trac tickets were found above: Ticket 1361 --> https://svn.open-mpi.org/trac/ompi/ticket/1361
2008-08-01 02:11:46 +04:00
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
/* do we want session output left open? */
mca_base_param_reg_int_name("orte", "leave_session_attached",
"Whether applications and/or daemons should leave their sessions "
"attached so that any output can be received - this allows X forwarding "
"without all the attendant debugging output",
false, false, (int)false, &value);
orte_leave_session_attached = OPAL_INT_TO_BOOL(value);
Ensure debug output levels if we are debugging This commit was SVN r26358.
2012-04-29 04:03:28 +04:00
/* if any debug level is set, ensure we output debug level dumps */
if (orte_debug_flag || orte_debug_daemons_flag || orte_leave_session_attached) {
orte_devel_level_output = true;
}
Create the debugger attach fifo unless the user requests that we periodically poll insteaad. This commit was SVN r25714.
2012-01-11 23:44:22 +04:00
/* See comment in orte/tools/orterun/orterun.c about this MCA
Fixes trac:1361: mainly add new internal MCA parameter that orterun will set when it launches under debuggers using the --debug option. This commit was SVN r19116. The following Trac tickets were found above: Ticket 1361 --> https://svn.open-mpi.org/trac/ompi/ticket/1361
2008-08-01 02:11:46 +04:00
param (this param is internal) */
mca_base_param_reg_int_name("orte",
"in_parallel_debugger",
"Whether the application is being debugged "
"in a parallel debugger (default: false)",
true, false, 0, &value);
orte_in_parallel_debugger = OPAL_INT_TO_BOOL(value);
My apologies for doing this outside of the usual time restrictions, but we need to get this in so we can make progress. Move the ORTE-level debugger code back into orterun and out of the ORTE library to resolve symbol conflicts. This commit was SVN r25713.
2012-01-11 19:53:09 +04:00
mca_base_param_reg_int_name("orte",
"output_debugger_proctable",
"Whether or not to output the debugger proctable after launch (default: false)",
false, false, 0, &value);
orte_debugger_dump_proctable = OPAL_INT_TO_BOOL(value);
mca_base_param_reg_string_name("orte", "debugger_test_daemon",
"Name of the executable to be used to simulate a debugger colaunch (relative or absolute path)",
false, false, NULL, &orte_debugger_test_daemon);
mca_base_param_reg_int_name("orte",
"debugger_test_attach",
"Test debugger colaunch after debugger attachment",
false, false, 0, &value);
orte_debugger_test_attach = OPAL_INT_TO_BOOL(value);
mca_base_param_reg_int_name("orte",
"debugger_check_rate",
"Set rate (in secs) for auto-detect of debugger attachment (0 => do not check)",
false, false, 0, &orte_debugger_check_rate);
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
mca_base_param_reg_int_name("orte", "do_not_launch",
"Perform all necessary operations to prepare to launch the application, but do not actually launch it",
false, false, (int)false, &value);
orte_do_not_launch = OPAL_INT_TO_BOOL(value);
mca_base_param_reg_int_name("orte", "daemon_spin",
"Have any orteds spin until we can connect a debugger to them",
false, false, (int)false, &value);
orted_spin_flag = OPAL_INT_TO_BOOL(value);
Cleanly handle the failed start of an orted, or its unexpected failure after start. This commit will allow mpirun to exit cleanly when this occurs, and does a best-effort attempt to cleanup the mess. However, it still has two unresolved issues that need to be eventually addressed: 1. it depends upon the ability of the native environment to alert us that the orted has died/failed to start. I have included that support for SLURM, but other environments need to be done. 2. for some yet-to-be-determined reason, the message that tells the remaining daemons to "die" isn't getting out of the RML, even though no obvious blockage is standing in the way. Work will continue on resolving that problem. For now, the orteds appear to be exiting on their own quite nicely when they see their HNP "lifeline" disappear. This represents the best-available fix for ticket #221 so I am closing that ticket at this time. This commit was SVN r18536.
2008-05-29 17:38:27 +04:00
mca_base_param_reg_int_name("orte", "daemon_fail",
"Have the specified orted fail after init for debugging purposes",
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
false, false, ORTE_VPID_INVALID, &orted_debug_failure);
Cleanly handle the failed start of an orted, or its unexpected failure after start. This commit will allow mpirun to exit cleanly when this occurs, and does a best-effort attempt to cleanup the mess. However, it still has two unresolved issues that need to be eventually addressed: 1. it depends upon the ability of the native environment to alert us that the orted has died/failed to start. I have included that support for SLURM, but other environments need to be done. 2. for some yet-to-be-determined reason, the message that tells the remaining daemons to "die" isn't getting out of the RML, even though no obvious blockage is standing in the way. Work will continue on resolving that problem. For now, the orteds appear to be exiting on their own quite nicely when they see their HNP "lifeline" disappear. This represents the best-available fix for ticket #221 so I am closing that ticket at this time. This commit was SVN r18536.
2008-05-29 17:38:27 +04:00
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
mca_base_param_reg_int_name("orte", "daemon_fail_delay",
"Have the specified orted fail after specified number of seconds (default: 0 => no delay)",
false, false, 0, &orted_debug_failure_delay);
mca_base_param_reg_int_name("orte", "startup_timeout",
"Milliseconds/daemon to wait for startup before declaring failed_to_start (default: 0 => do not check)",
false, false, 0, &orte_startup_timeout);
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
/* check for timing requests */
mca_base_param_reg_int_name("orte", "timing",
"Request that critical timing loops be measured",
false, false, (int)false, &value);
orte_timing = OPAL_INT_TO_BOOL(value);
Continue to refine the timing instrumentation to identify where launch time is being spent This commit was SVN r20244.
2009-01-12 22:12:58 +03:00
mca_base_param_reg_int_name("orte", "timing_details",
"Request that detailed timing data by reported",
false, false, (int)false, &value);
orte_timing_details = OPAL_INT_TO_BOOL(value);
if (orte_timing_details) {
/* ensure the timing flag is set too */
orte_timing = true;
}
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
if (ORTE_PROC_IS_HNP) {
Continue to improve timing instrumentation. Add ability to store timing data directly to a file instead of just to stdout. This commit was SVN r20229.
2009-01-08 17:27:52 +03:00
char *tmp;
mca_base_param_reg_string_name("orte", "timing_file",
"Name of the file where timing data is to be written (relative or absolute path)",
false, false, NULL, &tmp);
if (orte_timing && NULL == tmp) {
/* send the timing output to stdout */
orte_timing_output = stdout;
} else if (NULL != tmp) {
/* make sure the timing flag is set */
orte_timing = true;
/* send the output to the indicated file */
orte_timing_output = fopen(tmp, "w");
if (NULL == orte_timing_output) {
/* couldn't be opened */
opal_output(0, "File %s could not be opened", tmp);
orte_timing_output = stderr;
}
}
}
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
/* User-level debugger info string */
mca_base_param_reg_string_name("orte", "base_user_debugger",
"Sequence of user-level debuggers to search for in orterun",
false, false, "totalview @mpirun@ -a @mpirun_args@ : ddt -n @np@ -start @executable@ @executable_argv@ @single_app@ : fxp @mpirun@ -a @mpirun_args@", NULL);
mca_base_param_reg_int_name("orte", "abort_timeout",
"Max time to wait [in secs] before aborting an ORTE operation (default: 1sec)",
false, false, 1, &value);
orte_max_timeout = 1000000.0 * value; /* convert to usec */
mca_base_param_reg_int_name("orte", "timeout_step",
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
"Time to wait [in usecs/proc] before aborting an ORTE operation (default: 1000 usec/proc)",
false, false, 1000, &orte_timeout_usec_per_proc);
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
/* default hostfile */
mca_base_param_reg_string_name("orte", "default_hostfile",
In accordance with prior releases, we are supposed to default to looking at the openmpi-default-hostfile as a default hostfile. Restore that behavior, but ignore the file if it is empty. Allow the user to ignore any MCA param setting pointing to a default hostfile by setting the param to "none" (via cmd line or whatever) - this allows them to override a setting in the system default MCA param file. This commit was SVN r25851.
2012-02-01 21:40:44 +04:00
"Name of the default hostfile (relative or absolute path, \"none\" to ignore environmental or default MCA param setting)",
Allow there to be no default hostfile without generating an error This commit was SVN r25930.
2012-02-15 08:16:05 +04:00
false, false, NULL, &strval);
if (NULL == strval) {
/* nothing was given, so define the default */
Fix location of the default hostfile. Thanks to Götz Waschk for identifying the problem. This commit was SVN r26441.
2012-05-15 20:13:39 +04:00
asprintf(&orte_default_hostfile, "%s/openmpi-default-hostfile", opal_install_dirs.sysconfdir);
Allow there to be no default hostfile without generating an error This commit was SVN r25930.
2012-02-15 08:16:05 +04:00
/* flag that nothing was given */
orte_default_hostfile_given = false;
Fix typo - thanks Pascal This commit was SVN r26064.
2012-02-28 18:33:55 +04:00
} else if (0 == strcmp(strval, "none")) {
In accordance with prior releases, we are supposed to default to looking at the openmpi-default-hostfile as a default hostfile. Restore that behavior, but ignore the file if it is empty. Allow the user to ignore any MCA param setting pointing to a default hostfile by setting the param to "none" (via cmd line or whatever) - this allows them to override a setting in the system default MCA param file. This commit was SVN r25851.
2012-02-01 21:40:44 +04:00
orte_default_hostfile = NULL;
Allow there to be no default hostfile without generating an error This commit was SVN r25930.
2012-02-15 08:16:05 +04:00
/* flag that it was given */
orte_default_hostfile_given = true;
} else {
orte_default_hostfile = strval;
/* flag that it was given */
orte_default_hostfile_given = true;
In accordance with prior releases, we are supposed to default to looking at the openmpi-default-hostfile as a default hostfile. Restore that behavior, but ignore the file if it is empty. Allow the user to ignore any MCA param setting pointing to a default hostfile by setting the param to "none" (via cmd line or whatever) - this allows them to override a setting in the system default MCA param file. This commit was SVN r25851.
2012-02-01 21:40:44 +04:00
}
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
Use this option only on Windows, so protect it with #ifdef __WINDOWS__. This commit was SVN r22701.
2010-02-24 11:50:03 +03:00
#ifdef __WINDOWS__
Add a global option for inputing the head node name for Windows CCP trough command line. This commit was SVN r22688.
2010-02-23 22:42:51 +03:00
mca_base_param_reg_string_name("orte", "ccp_headnode",
Add an mca param to indicate if orte progress threads are to be enabled. Error out if this is given and libevent thread support was not built. This commit was SVN r24089.
2010-11-25 02:28:00 +03:00
"Name of the cluster head node. (For Windows CCP only.)",
false, false,
NULL, &orte_ccp_headnode);
Use this option only on Windows, so protect it with #ifdef __WINDOWS__. This commit was SVN r22701.
2010-02-24 11:50:03 +03:00
#endif
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
Fix some major bit-rot on scalable launch. If static ports are provided, then daemons can connect back to the HNP via the routed connection tree instead of doing so directly. In order to do that at scale, the node list must be passed as a regular expression - otherwise, the orted command line gets too long. Over the course of time, usage of static ports got corrupted in several places, the "parent" info got incorrectly reset, etc. So correct all that and get the regex-based wireup going again. Also, don't pass node lists if static ports aren't enabled - they are of no value to the orted and just create the possibility of overly-long cmd lines. This commit was SVN r24860.
2011-07-07 22:54:30 +04:00
/* regex of nodes in system */
mca_base_param_reg_string_name("orte", "node_regex",
"Regular expression defining nodes in the system",
false, false, NULL, &orte_node_regex);
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
/* whether or not to keep FQDN hostnames */
mca_base_param_reg_int_name("orte", "keep_fqdn_hostnames",
"Whether or not to keep FQDN hostnames [default: no]",
false, false, (int)false, &value);
orte_keep_fqdn_hostnames = OPAL_INT_TO_BOOL(value);
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
/* whether to tag output */
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
mca_base_param_reg_int_name("orte", "tag_output",
"Tag all output with [job,rank] (default: false)",
false, false, (int) false, &value);
orte_tag_output = OPAL_INT_TO_BOOL(value);
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
/* if we requested xml output, be sure to tag the output as well */
if (orte_xml_output) {
orte_tag_output = true;
}
Initial work on xml support - allocation and job map outputs completed. More to come. This commit was SVN r18587.
2008-06-05 00:53:12 +04:00
mca_base_param_reg_int_name("orte", "xml_output",
"Display all output in XML format (default: false)",
false, false, (int) false, &value);
orte_xml_output = OPAL_INT_TO_BOOL(value);
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
mca_base_param_reg_string_name("orte", "xml_file",
"Provide all output in XML format to the specified file",
false, false, NULL, &strval);
if (NULL != strval) {
if (ORTE_PROC_IS_HNP && NULL == orte_xml_fp) {
/* only the HNP opens this file! Make sure it only happens once */
orte_xml_fp = fopen(strval, "w");
if (NULL == orte_xml_fp) {
opal_output(0, "Could not open specified xml output file: %s", strval);
return ORTE_ERROR;
}
}
/* ensure we set the flags to tag output */
orte_xml_output = true;
Implement the xml formatted output of stdout/err/diag. Force -tag-output if -xml is set. This commit was SVN r20302.
2009-01-20 19:58:31 +03:00
orte_tag_output = true;
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
} else {
/* default to stdout */
orte_xml_fp = stdout;
Implement the xml formatted output of stdout/err/diag. Force -tag-output if -xml is set. This commit was SVN r20302.
2009-01-20 19:58:31 +03:00
}
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
/* whether to timestamp output */
mca_base_param_reg_int_name("orte", "timestamp_output",
"Timestamp all application process output (default: false)",
false, false, (int) false, &value);
orte_timestamp_output = OPAL_INT_TO_BOOL(value);
/* redirect output into files */
mca_base_param_reg_string_name("orte", "output_filename",
"Redirect output from application processes into filename.rank [default: NULL]",
false, false, NULL, &orte_output_filename);
Per request from IBM/Eclipse, provide MCA param to request output when nodes are resolved to a different nodename. This really only happens for the node that mpirun executes on, but they need the alert so they can do string matching of node names. This commit was SVN r20032.
2008-11-24 22:57:08 +03:00
mca_base_param_reg_int_name("orte", "show_resolved_nodenames",
"Display any node names that are resolved to a different name (default: false)",
false, false, (int) false, &value);
orte_show_resolved_nodenames = OPAL_INT_TO_BOOL(value);
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
mca_base_param_reg_int_name("orte", "hetero_apps",
"Indicates that multiple app_contexts are being provided that are a mix of 32/64 bit binaries (default: false)",
false, false, (int) false, &value);
orte_hetero_apps = OPAL_INT_TO_BOOL(value);
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#if OPAL_HAVE_HWLOC
Minimize the amount of topology info returned by the daemons. Most clusters, especially at scale, use the same node topology on every node, so there is no re ason to return the topology from every daemon. Borrow a page from the --hetero-apps page and let users indicate that the node topology differs by adding a -- hetero-nodes option to mpirun. If the option is set, then every daemon returns topology info. If not set, then only daemon vpid=1 returns it. We always want one daemon to return the topology as the head node is often different from the compute nodes. Having one daemon return the compute node topolo gy allows us to detect any such difference. All compute nodes are then set to the same topology. This commit was SVN r25408.
2011-11-01 22:43:10 +04:00
mca_base_param_reg_int_name("orte", "hetero_nodes",
"Nodes in cluster may differ in topology, so send the topology back from each node [Default = false]",
false, false, (int) false, &value);
orte_hetero_nodes = OPAL_INT_TO_BOOL(value);
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#endif
Per the July technical meeting: Standardize the handling of the orte launch agent option across PLMs. This has been a consistent complaint I have received - each PLM would register its own MCA param to get input on the launch agent for remote nodes (in fact, one or two didn't, but most did). This would then get handled in various and contradictory ways. Some PLMs would accept only a one-word input. Others accepted multi-word args such as "valgrind orted", but then some would error by putting any prefix specified on the cmd line in front of the incorrect argument. For example, while using the rsh launcher, if you specified "valgrind orted" as your launch agent and had "--prefix foo" on you cmd line, you would attempt to execute "ssh foo/valgrind orted" - which obviously wouldn't work. This was all -very- confusing to users, who had to know which PLM was being used so they could even set the right mca param in the first place! And since we don't warn about non-recognized or non-used mca params, half of the time they would wind up not doing what they thought they were telling us to do. To solve this problem, we did the following: 1. removed all mca params from the individual plms for the launch agent 2. added a new mca param "orte_launch_agent" for this purpose. To further simplify for users, this comes with a new cmd line option "--launch-agent" that can take a multi-word string argument. The value of the param defaults to "orted". 3. added a PLM base function that processes the orte_launch_agent value and adds the contents to a provided argv array. This can subsequently be harvested at-will to handle multi-word values 4. modified the PLMs to use this new function. All the PLMs except for the rsh PLM required very minor change - just called the function and moved on. The rsh PLM required much larger changes as - because of the rsh/ssh cmd line limitations - we had to correctly prepend any provided prefix to the correct argv entry. 5. added a new opal_argv_join_range function that allows the caller to "join" argv entries between two specified indices Please let me know of any problems. I tried to make this as clean as possible, but cannot compile all PLMs to ensure all is correct. This commit was SVN r19097.
2008-07-30 22:26:24 +04:00
/* allow specification of the launch agent */
mca_base_param_reg_string_name("orte", "launch_agent",
"Command used to start processes on remote nodes (default: orted)",
false, false, "orted", &orte_launch_agent);
Establish an MCA param "orte_allocation_required" so that a system can require the user have an RM-provided allocation in order to run. This helps prevent the problem where a user forgets to get an allocation on an RM-managed cluster, and then executes mpirun on the head node - thus causing all of their mpi procs to launch on the head node, usually bringing it to its knees. Since OMPI allows mpirun to default to the local node, and since users want to retain the option to co-locate procs with mpirun, we needed another param to block this error case. This commit was SVN r19135.
2008-08-04 18:25:19 +04:00
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
mca_base_param_reg_string_name("orte", "fork_agent",
"Command used to fork processes on remote nodes (default: NULL)",
false, false, NULL, &strval);
if (NULL != strval) {
orte_fork_agent = opal_argv_split(strval, ' ');
}
Establish an MCA param "orte_allocation_required" so that a system can require the user have an RM-provided allocation in order to run. This helps prevent the problem where a user forgets to get an allocation on an RM-managed cluster, and then executes mpirun on the head node - thus causing all of their mpi procs to launch on the head node, usually bringing it to its knees. Since OMPI allows mpirun to default to the local node, and since users want to retain the option to co-locate procs with mpirun, we needed another param to block this error case. This commit was SVN r19135.
2008-08-04 18:25:19 +04:00
/* whether or not to require RM allocation */
mca_base_param_reg_int_name("orte", "allocation_required",
"Whether or not an allocation by a resource manager is required [default: no]",
false, false, (int)false, &value);
orte_allocation_required = OPAL_INT_TO_BOOL(value);
Add the ability to forward SIGTSTP (converted to SIGSTOP) and SIGCONT to the a.outs. By default, they are not forwarded and the behavior remains as it has always been. However, if one runs with --mca orte_forward_job_control 1, then mpirun will catch those two signals and forward them to the orteds which will deliver them to the a.outs. We have had requests for this feature. This commit was SVN r20391.
2009-01-30 21:50:10 +03:00
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
/* generate new terminal windows to display output from specified ranks */
mca_base_param_reg_string_name("orte", "xterm",
"Create a new xterm window and display output from the specified ranks there [default: none]",
false, false, NULL, &orte_xterm);
Fix the case where a user supplies the -xterm option, which requires that we leave ssh sessions attached. This commit was SVN r24668.
2011-05-02 16:39:55 +04:00
if (NULL != orte_xterm) {
/* if an xterm request is given, we have to leave any ssh
* sessions attached so the xterm window manager can get
* back to the controlling terminal
*/
orte_leave_session_attached = true;
Per request from Brian, enable mapping of stddiag output (output from opal_output calls) to stderr of the local process. This allows you to obtain that output in a local window (for example, when using xterm for each process) instead of having it automatically forwarded to mpirun. Turn this on automatically whenever someone uses the -xterm option, and to be set manually using the orte_map_stddiag_to_stderr mca param. This commit was SVN r26352.
2012-04-27 18:39:34 +04:00
/* also want to redirect stddiag output from opal_output
* to stderr from the process so those messages show
* up in the xterm window instead of being forwarded to mpirun
*/
orte_map_stddiag_to_stderr = true;
Fix the case where a user supplies the -xterm option, which requires that we leave ssh sessions attached. This commit was SVN r24668.
2011-05-02 16:39:55 +04:00
}
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
Per request from Brian, enable mapping of stddiag output (output from opal_output calls) to stderr of the local process. This allows you to obtain that output in a local window (for example, when using xterm for each process) instead of having it automatically forwarded to mpirun. Turn this on automatically whenever someone uses the -xterm option, and to be set manually using the orte_map_stddiag_to_stderr mca param. This commit was SVN r26352.
2012-04-27 18:39:34 +04:00
/* whether or not to map stddiag to stderr */
mca_base_param_reg_int_name("orte", "map_stddiag_to_stderr",
"Map output from opal_output to stderr of the local process [default: no]",
false, false,
(int) false, &value);
orte_map_stddiag_to_stderr = OPAL_INT_TO_BOOL(value);
Add the ability to forward SIGTSTP (converted to SIGSTOP) and SIGCONT to the a.outs. By default, they are not forwarded and the behavior remains as it has always been. However, if one runs with --mca orte_forward_job_control 1, then mpirun will catch those two signals and forward them to the orteds which will deliver them to the a.outs. We have had requests for this feature. This commit was SVN r20391.
2009-01-30 21:50:10 +03:00
/* whether or not to forward SIGTSTP and SIGCONT signals */
mca_base_param_reg_int_name("orte", "forward_job_control",
"Forward SIGTSTP (after converting to SIGSTOP) and SIGCONT signals to the application procs [default: no]",
false, false,
(int) false, &value);
orte_forward_job_control = OPAL_INT_TO_BOOL(value);
Enable comm_spawn of slave processes, currently only active for the rsh, slurm, and tm environments. Establish support for local rsh environments in the plm/base so that rsh of local slaves can be done by any environment that supports it. Create new orte_rsh_agent param so users can specify rsh agent from outside of rsh plm, and sym link that to the old plm_rsh_agent and pls_rsh_agent options. Modify the orte-bootproxy to pass prefix for the remote slave to support hetero/hybrid scenarios This commit was SVN r20492.
2009-02-09 23:44:44 +03:00
Provide a "progress meter" for launch that outputs progress as we are launching, especially on large jobs. Also, provide a timeout mechanism so that we cleanly abort if we don't get a response from the next daemon in a specified time. This commit was SVN r21359.
2009-06-03 03:52:02 +04:00
/* whether or not to report launch progress */
mca_base_param_reg_int_name("orte", "report_launch_progress",
"Output a brief periodic report on launch progress [default: no]",
false, false,
(int) false, &value);
orte_report_launch_progress = OPAL_INT_TO_BOOL(value);
if (orte_report_launch_progress) {
/* ensure the startup timeout is set to something reasonable */
if (0 == orte_startup_timeout) {
orte_startup_timeout = 2000; /* default to 2 seconds */
}
}
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
Added new solaris sysinfo module. Also added code to assign orte_local_chip_type and orte_local_chip_model in MPI processes it the appropriate sysinfo module found the values on the machine. This commit was SVN r23581.
2010-08-09 23:28:56 +04:00
/* cluster hardware info detected by orte only */
mca_base_param_reg_string_name("orte", "cpu_type",
corrected a couple places in orte where it said cpu_model when it should have been cpu_type. This commit was SVN r24221.
2011-01-11 22:56:26 +03:00
"cpu type detected in node",
Added new solaris sysinfo module. Also added code to assign orte_local_chip_type and orte_local_chip_model in MPI processes it the appropriate sysinfo module found the values on the machine. This commit was SVN r23581.
2010-08-09 23:28:56 +04:00
true, false, NULL, &orte_local_cpu_type);
mca_base_param_reg_string_name("orte", "cpu_model",
"cpu model detected in node",
true, false, NULL, &orte_local_cpu_model);
Enable a new command line option to --report-events that instructs mpirun to RML-report specific events during job life to the requestor. This commit was SVN r21954.
2009-09-09 09:28:45 +04:00
/* tool communication controls */
mca_base_param_reg_string_name("orte", "report_events",
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
"URI to which events are to be reported (default: NULL)",
Enable a new command line option to --report-events that instructs mpirun to RML-report specific events during job life to the requestor. This commit was SVN r21954.
2009-09-09 09:28:45 +04:00
false, false, NULL, &orte_report_events_uri);
if (NULL != orte_report_events_uri) {
orte_report_events = true;
}
The ability to add procs to a running job was unfortunately borked when we added the detection of a proc exiting before calling init. Re-enable it here, ensuring that procs that are being restarted and/or added to a job do -not- call barrier during orte_init. This commit was SVN r22404.
2010-01-14 20:59:42 +03:00
/* barrier control */
mca_base_param_reg_int_name("orte", "do_not_barrier",
"Do not barrier in orte_init",
true, false,
(int) false, &value);
orte_do_not_barrier = OPAL_INT_TO_BOOL(value);
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
mca_base_param_reg_int_name("orte", "enable_recovery",
"Enable recovery from process failure [Default = disabled]",
false, false,
(int)false, &value);
orte_enable_recovery = OPAL_INT_TO_BOOL(value);
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
mca_base_param_reg_int_name("orte", "max_restarts",
"Max number of times to restart a failed process",
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
false, false,
A max_restarts value of -1 is now valid and indicates infinite restarts, so correct the validity check This commit was SVN r24546.
2011-03-20 04:17:00 +03:00
0, &orte_max_restarts);
if (!orte_enable_recovery && orte_max_restarts != 0) {
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
if (ORTE_PROC_IS_HNP) {
opal_output(orte_clean_output,
"------------------------------------------------------------------\n"
Some relatively minor C/R related cleanup * Fix a configure warning for checking --enable-ft-thread * In hnp and orted ErrMgr components check to see if other components have already recovered this process before trying to recover it again. * Fix 'npernode' for restarting using the resilient rmaps component * export ompi_info_set, so that internal functionality can use it. This commit was SVN r23535.
2010-07-30 22:59:34 +04:00
"The MCA param orte_enable_recovery was not set to true, but\n"
A max_restarts value of -1 is now valid and indicates infinite restarts, so correct the validity check This commit was SVN r24546.
2011-03-20 04:17:00 +03:00
"a value was provided for the number of restarts:\n\n"
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
"Max restarts: %d\n"
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
"We are enabling process recovery and continuing execution. To avoid\n"
Some relatively minor C/R related cleanup * Fix a configure warning for checking --enable-ft-thread * In hnp and orted ErrMgr components check to see if other components have already recovered this process before trying to recover it again. * Fix 'npernode' for restarting using the resilient rmaps component * export ompi_info_set, so that internal functionality can use it. This commit was SVN r23535.
2010-07-30 22:59:34 +04:00
"this warning in the future, please set the orte_enable_recovery\n"
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
"param to non-zero.\n"
"------------------------------------------------------------------",
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
orte_max_restarts);
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
}
orte_enable_recovery = true;
}
Expand our handling of non-zero exit status. If a process exits with non-zero status, pass that info along to the user in case it means something to them, even if the process also exited without calling MPI_Finalize. If the process calls MPI_Abort, that trumps the exit status question. Provide a new MCA param that allows the user to direct that we abort the job once a process exits with non-zero status. No recovery is allowed in such cases to avoid trying to restart a process that has already exited MPI. This commit was SVN r24614.
2011-04-14 19:04:21 +04:00
mca_base_param_reg_int_name("orte", "abort_on_non_zero_status",
"Abort the job if any process returns a non-zero exit status - no restart in such cases",
Change the default to "abort" the job when any process exits with a non-zero status. Add the required code to ensure the orted tells the HNP about the problem. This commit was SVN r26270.
2012-04-14 01:19:46 +04:00
false, false, (int)true, &value);
Expand our handling of non-zero exit status. If a process exits with non-zero status, pass that info along to the user in case it means something to them, even if the process also exited without calling MPI_Finalize. If the process calls MPI_Abort, that trumps the exit status question. Provide a new MCA param that allows the user to direct that we abort the job once a process exits with non-zero status. No recovery is allowed in such cases to avoid trying to restart a process that has already exited MPI. This commit was SVN r24614.
2011-04-14 19:04:21 +04:00
orte_abort_non_zero_exit = OPAL_INT_TO_BOOL(value);
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
mca_base_param_reg_int_name("orte", "report_child_jobs_separately",
"Return the exit status of the primary job only",
false, false,
(int)false, &value);
orte_report_child_jobs_separately = OPAL_INT_TO_BOOL(value);
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
mca_base_param_reg_int_name("orte", "child_time_to_exit",
"Max time a spawned child job is allowed to run after the primary job has terminated (seconds)",
false, false,
INT_MAX, &value);
orte_child_time_to_exit.tv_sec = value;
orte_child_time_to_exit.tv_usec = 0;
Add an mca param to indicate if orte progress threads are to be enabled. Error out if this is given and libevent thread support was not built. This commit was SVN r24089.
2010-11-25 02:28:00 +03:00
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
mca_base_param_reg_int_name("orte", "stat_history_size",
"Number of stat samples to keep",
false, false, 1, &orte_stat_history_size);
Remove stale orte_vm_launch param. Add a param that allows users to specify envars to forward/set so they can do it in the MCA param file instead of only via mpirun cmd line. This commit was SVN r25580.
2011-12-07 01:31:22 +04:00
mca_base_param_reg_string_name("orte", "forward_envars",
"Comma-delimited environmental variables to forward, can include value to set",
false, false, NULL, &orte_forward_envars);
Make preload_binaries an MCA param so it can be set in the default MCA parameters for a system This commit was SVN r25728.
2012-01-17 21:16:05 +04:00
mca_base_param_reg_int_name("orte", "preload_binaries",
"Preload the binaries on remote machines before starting remote proceses",
Fix the default This commit was SVN r25733.
2012-01-18 01:09:27 +04:00
false, false, (int)false, &value);
Make preload_binaries an MCA param so it can be set in the default MCA parameters for a system This commit was SVN r25728.
2012-01-17 21:16:05 +04:00
orte_preload_binaries = OPAL_INT_TO_BOOL(value);
Allow partial use of allocations by specifying the max number of daemons (i.e., max VM size) for the job This commit was SVN r26499.
2012-05-27 20:48:19 +04:00
mca_base_param_reg_int_name("orte", "max_vm_size",
"Maximum size of virtual machine - used to subdivide allocation",
false, false, -1, &orte_max_vm_size);
Ensure that we don't attempt to use common ports if --disable-static was specified. This commit was SVN r26620.
2012-06-20 07:14:11 +04:00
#if ORTE_ENABLE_STATIC_PORTS
Improve launch performance on clusters that use dedicated nodes by instructing the orteds to use the same port as the HNP, thus allowing them to "rollup" their initial callback via the routed network. This substantially reduces the HNP bottleneck and the number of ports opened by the HNP. Restore enable-static-ports option by default - the Cray will have to disable it to get around their library issues, but that's just a warning problem as opposed to blocking the build. This commit was SVN r26606.
2012-06-15 14:15:07 +04:00
mca_base_param_reg_int_name("orte", "use_common_port",
"Daemons use same port as HNP",
Turn off the common port by default by now until we get rollup working properly on ALL platforms This commit was SVN r27060.
2012-08-16 02:13:04 +04:00
false, false, (int)false, &value);
Improve launch performance on clusters that use dedicated nodes by instructing the orteds to use the same port as the HNP, thus allowing them to "rollup" their initial callback via the routed network. This substantially reduces the HNP bottleneck and the number of ports opened by the HNP. Restore enable-static-ports option by default - the Cray will have to disable it to get around their library issues, but that's just a warning problem as opposed to blocking the build. This commit was SVN r26606.
2012-06-15 14:15:07 +04:00
orte_use_common_port = OPAL_INT_TO_BOOL(value);
Ensure that we don't attempt to use common ports if --disable-static was specified. This commit was SVN r26620.
2012-06-20 07:14:11 +04:00
#else
orte_use_common_port = false;
#endif
Improve launch performance on clusters that use dedicated nodes by instructing the orteds to use the same port as the HNP, thus allowing them to "rollup" their initial callback via the routed network. This substantially reduces the HNP bottleneck and the number of ports opened by the HNP. Restore enable-static-ports option by default - the Cray will have to disable it to get around their library issues, but that's just a warning problem as opposed to blocking the build. This commit was SVN r26606.
2012-06-15 14:15:07 +04:00
Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm. Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed. This commit was SVN r18664.
2008-06-18 07:15:56 +04:00
#endif /* ORTE_DISABLE_FULL_SUPPORT */
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
return ORTE_SUCCESS;
}
Копировать постоянную ссылку