ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
f7d3234f33
openmpi/orte/tools/orterun/orterun.c

2119 строки
79 KiB
C
Исходник Обычный вид История

* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* -*- C -*-
*
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
Update the copyright notices for IU and UTK. This commit was SVN r7999.
2005-11-05 22:57:48 +03:00
* University Research and Technology
* Corporation. All rights reserved.
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
* Copyright (c) 2004-2008 The University of Tennessee and The University
Update the copyright notices for IU and UTK. This commit was SVN r7999.
2005-11-05 22:57:48 +03:00
* of Tennessee Research Foundation. All rights
* reserved.
Move the error name macro to the errmgr framework. Add a second level of tracing. Remove an obsolete file. This commit was SVN r7445.
2005-09-20 21:09:11 +04:00
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
* University of Stuttgart. All rights reserved.
Add UC copyright This commit was SVN r5009.
2005-03-24 15:43:37 +03:00
* Copyright (c) 2004-2005 The Regents of the University of California.
* All rights reserved.
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
* Copyright (c) 2006-2011 Cisco Systems, Inc. All rights reserved.
Add the ability to forward SIGTSTP (converted to SIGSTOP) and SIGCONT to the a.outs. By default, they are not forwarded and the behavior remains as it has always been. However, if one runs with --mca orte_forward_job_control 1, then mpirun will catch those two signals and forward them to the orteds which will deliver them to the a.outs. We have had requests for this feature. This commit was SVN r20391.
2009-01-30 21:50:10 +03:00
* Copyright (c) 2007-2009 Sun Microsystems, Inc. All rights reserved.
OS X apparently really doesn't like shared libraries with unresolvable symbols in them and environ is defined only in the final application (probably in crt1.o). Apple provides a function for getting at the environment, so use that instead if it's available. This commit was SVN r14857.
2007-06-05 07:03:59 +04:00
* Copyright (c) 2007 Los Alamos National Security, LLC. All rights
* reserved.
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
* $COPYRIGHT$
Move the error name macro to the errmgr framework. Add a second level of tracing. Remove an obsolete file. This commit was SVN r7445.
2005-09-20 21:09:11 +04:00
*
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
* Additional copyrights may follow
Move the error name macro to the errmgr framework. Add a second level of tracing. Remove an obsolete file. This commit was SVN r7445.
2005-09-20 21:09:11 +04:00
*
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
* $HEADER$
*/
#include "orte_config.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/constants.h"
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
- Get pgcc on XT to complain less: In case we use memcmp, strlen, strup and friends include <string.h> Also several constants.h are not included directly - Let's have mca_topo_base_cart_create return ompi-errors in ompi/mca/topo/base/topo_base_cart_create.c This commit was SVN r20773.
2009-03-13 05:10:32 +03:00
#ifdef HAVE_STRING_H
#include <string.h>
#endif
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
#include <stdio.h>
#ifdef HAVE_UNISTD_H
#include <unistd.h>
#endif
#ifdef HAVE_SYS_PARAM_H
#include <sys/param.h>
#endif
#include <errno.h>
#include <signal.h>
#include <ctype.h>
Protect the includes. This commit was SVN r8532.
2005-12-18 01:05:10 +03:00
#ifdef HAVE_SYS_TYPES_H
* add some missing header files so that it compiles on OS X This commit was SVN r5123.
2005-04-01 04:30:37 +04:00
#include <sys/types.h>
Protect the includes. This commit was SVN r8532.
2005-12-18 01:05:10 +03:00
#endif /* HAVE_SYS_TYPES_H */
#ifdef HAVE_SYS_WAIT_H
* add some missing header files so that it compiles on OS X This commit was SVN r5123.
2005-04-01 04:30:37 +04:00
#include <sys/wait.h>
Protect the includes. This commit was SVN r8532.
2005-12-18 01:05:10 +03:00
#endif /* HAVE_SYS_WAIT_H */
Bring over the rest of the ctrl-c fixes. This commit includes: 1. add a "cancel_operation" API to the pls components that allows orterun to demand that an orted operation (e.g., terminate_job) be immediately cancelled and abandoned. 2. changes the pls orted commands from blocking to non-blocking. This allows us to interrupt those operations should an orted be non-responsive. The change also adds an orte_abort_timeout that limits how long orterun will automatically wait for the orteds to respond - if the terminate command, for example, doesn't see orted response within that time, then we printout an appropriate error message and just give up. 3. modifies orterun to allow multiple ctrl-c's to simply abort the program even if the orteds have not responded 4. does some cleanup on the orte-level mca params so that their implementation looks a lot more like that of ompi - makes it easier to maintain. This change also includes the definition of an orte_abort_timeout struct and associated MCA param (can't have too many!) so you can set the time after which orterun gives up on waiting for orteds to respond This needs more testing before migrating to 1.2. This commit was SVN r13304.
2007-01-25 17:17:44 +03:00
#ifdef HAVE_SYS_TIME_H
#include <sys/time.h>
Deal with the include spree. Protect "environ" on Windows. Some others minors modifications in order to make it compile [again] on Windows. This commit was SVN r14188.
2007-04-01 20:16:54 +04:00
#endif /* HAVE_SYS_TIME_H */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Update libevent to the 2.0 series, currently at 2.0.7rc. We will update to their final release when it becomes available. Currently known errors exist in unused portions of the libevent code. This revision passes the IBM test suite on a Linux machine and on a standalone Mac. This is a fairly intrusive change, but outside of the moving of opal/event to opal/mca/event, the only changes involved (a) changing all calls to opal_event functions to reflect the new framework instead, and (b) ensuring that all opal_event_t objects are properly constructed since they are now true opal_objects. Note: Shiqing has just returned from vacation and has not yet had a chance to complete the Windows integration. Thus, this commit almost certainly breaks Windows support on the trunk. However, I want this to have a chance to soak for as long as possible before I become less available a week from today (going to be at a class for 5 days, and thus will only be sparingly available) so we can find and fix any problems. Biggest change is moving the libevent code from opal/event to a new opal/mca/event framework. This was done to make it much easier to update libevent in the future. New versions can be inserted as a new component and tested in parallel with the current version until validated, then we can remove the earlier version if we so choose. This is a statically built framework ala installdirs, so only one component will build at a time. There is no selection logic - the sole compiled component simply loads its function pointers into the opal_event struct. I have gone thru the code base and converted all the libevent calls I could find. However, I cannot compile nor test every environment. It is therefore quite likely that errors remain in the system. Please keep an eye open for two things: 1. compile-time errors: these will be obvious as calls to the old functions (e.g., opal_evtimer_new) must be replaced by the new framework APIs (e.g., opal_event.evtimer_new) 2. run-time errors: these will likely show up as segfaults due to missing constructors on opal_event_t objects. It appears that it became a typical practice for people to "init" an opal_event_t by simply using memset to zero it out. This will no longer work - you must either OBJ_NEW or OBJ_CONSTRUCT an opal_event_t. I tried to catch these cases, but may have missed some. Believe me, you'll know when you hit it. There is also the issue of the new libevent "no recursion" behavior. As I described on a recent email, we will have to discuss this and figure out what, if anything, we need to do. This commit was SVN r23925.
2010-10-24 22:35:54 +04:00
#include "opal/mca/event/event.h"
Merge the /tmp/jms-installdirs-trunk branch into the trunk. This finally brings in functionality that is already on the 1.2 branch, and was developed and tested in the v1.2ofed branch (and other places). Short version of new features: * Support for ibv_fork_init() * Automatically fill in the openib BTL bandwidth value by querying the HCA port * Installdirs functionality * Fixes to always use -I in the Fortran wrapper compilers (#924) * Gleb's mpool updates * Remove some kruft in btl/openib/configure.m4, therefore fixing the harmless warnings noted in #665 * Bunches of updates to the Linux RPM spec file I.e., effectively the same thing that r14411 brought to the v1.2 branch. Also effectively brought in r14432 and r14433 (some fixes on top of the original r14411 commit to v1.2). Still need to bring in the moral equivalent of r14445 after this commit (fixes to installdirs). This commit was SVN r14449. The following SVN revision numbers were found above: r14411 --> open-mpi/ompi@83b31314ae25efa4a03dfc560b569322a18e3d23 r14432 --> open-mpi/ompi@a48f160595abf35f5826e9370135a9056d4da486 r14433 --> open-mpi/ompi@68f346d2bc482ae29df0cdfe82171dcdb28e4a01 r14445 --> open-mpi/ompi@13d366b827719ec7078db17315513d5c6291ab96
2007-04-21 04:15:05 +04:00
#include "opal/mca/installdirs/installdirs.h"
Warn users if they set opal_paffinity_alone, either to true or false, that this parameter is no longer functional - they must use the --bind-to option and its corresponding mca param. This commit was SVN r25567.
2011-12-03 05:10:52 +04:00
#include "opal/mca/paffinity/base/base.h"
Add trace to the daemons and orterun so we can tell when their callbacks are being exercised. This commit was SVN r7432.
2005-09-19 21:20:01 +04:00
#include "opal/mca/base/base.h"
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
#include "opal/util/argv.h"
- On the way to get the BTLs split out and lessen dependency on orte: Often, orte/util/show_help.h is included, although no functionality is required -- instead, most often opal_output.h, or orte/mca/rml/rml_types.h Please see orte_show_help_replacement.sh commited next. - Local compilation (Linux/x86_64) w/ -Wimplicit-function-declaration actually showed two *missing* #include "orte/util/show_help.h" in orte/mca/odls/base/odls_base_default_fns.c and in orte/tools/orte-top/orte-top.c Manually added these. Let's have MTT the last word. This commit was SVN r20557.
2009-02-14 05:26:12 +03:00
#include "opal/util/output.h"
Add trace to the daemons and orterun so we can tell when their callbacks are being exercised. This commit was SVN r7432.
2005-09-19 21:20:01 +04:00
#include "opal/util/basename.h"
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
#include "opal/util/cmd_line.h"
Add trace to the daemons and orterun so we can tell when their callbacks are being exercised. This commit was SVN r7432.
2005-09-19 21:20:01 +04:00
#include "opal/util/opal_environ.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "opal/util/opal_getcwd.h"
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
#include "orte/util/show_help.h"
Utilize Tim M's suggestion and use atomics to do the locking. This commit was SVN r17767.
2008-03-07 00:36:32 +03:00
#include "opal/sys/atomic.h"
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Merging in the jjhursey-ft-cr-stable branch (r13912 : HEAD). This merge adds Checkpoint/Restart support to Open MPI. The initial frameworks and components support a LAM/MPI-like implementation. This commit follows the risk assessment presented to the Open MPI core development group on Feb. 22, 2007. This commit closes trac:158 More details to follow. This commit was SVN r14051. The following SVN revisions from the original message are invalid or inconsistent and therefore were not cross-referenced: r13912 The following Trac tickets were found above: Ticket 158 --> https://svn.open-mpi.org/trac/ompi/ticket/158
2007-03-17 02:11:45 +03:00
#include "opal/runtime/opal_cr.h"
#endif
Fix for ticket #91 mpirun/orterun now has an option to print the version number. If -V/--version is given, it will print the version number. If it's the only option, we exit cleanly. Otherwise, we continue on as if --version wasn't given (except we've printed the version number). --This line, and th se below, will be ignored-- M orte/tools/orterun/orterun.c M orte/tools/orterun/help-orterun.txt This commit was SVN r10276.
2006-06-09 21:21:23 +04:00
#include "opal/version.h"
Merge the /tmp/jms-installdirs-trunk branch into the trunk. This finally brings in functionality that is already on the 1.2 branch, and was developed and tested in the v1.2ofed branch (and other places). Short version of new features: * Support for ibv_fork_init() * Automatically fill in the openib BTL bandwidth value by querying the HCA port * Installdirs functionality * Fixes to always use -I in the Fortran wrapper compilers (#924) * Gleb's mpool updates * Remove some kruft in btl/openib/configure.m4, therefore fixing the harmless warnings noted in #665 * Bunches of updates to the Linux RPM spec file I.e., effectively the same thing that r14411 brought to the v1.2 branch. Also effectively brought in r14432 and r14433 (some fixes on top of the original r14411 commit to v1.2). Still need to bring in the moral equivalent of r14445 after this commit (fixes to installdirs). This commit was SVN r14449. The following SVN revision numbers were found above: r14411 --> open-mpi/ompi@83b31314ae25efa4a03dfc560b569322a18e3d23 r14432 --> open-mpi/ompi@a48f160595abf35f5826e9370135a9056d4da486 r14433 --> open-mpi/ompi@68f346d2bc482ae29df0cdfe82171dcdb28e4a01 r14445 --> open-mpi/ompi@13d366b827719ec7078db17315513d5c6291ab96
2007-04-21 04:15:05 +04:00
#include "opal/runtime/opal.h"
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
#include "opal/util/os_path.h"
Improve the efficiency of the launch system by changing the outer loop to being over app_context, and adding a flag to the app_context so the daemon can record that "this app is on my node" when decoding the launch msg. If the --wdir option is given, check to see if the user provided a relative path. If so, convert it to an absolute path. This is needed to maintain consistent behavior across environements. Some environments automatically chdir to your current working directory when launching the remote orted, while others (e.g., ssh) don't. This levels the playing field and reduces user surprise. This commit was SVN r20342.
2009-01-25 15:39:24 +03:00
#include "opal/util/path.h"
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
#include "opal/class/opal_pointer_array.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "opal/dss/dss.h"
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
Add trace to the daemons and orterun so we can tell when their callbacks are being exercised. This commit was SVN r7432.
2005-09-19 21:20:01 +04:00
#include "orte/util/proc_info.h"
Push a generated "key" out to all the processes. This is necessary for some interconnect wireup in which all processes must agree on a "key" to initialize the interconnect with. This commit was SVN r11653.
2006-09-14 19:27:17 +04:00
#include "orte/util/pre_condition_transports.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/util/session_dir.h"
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
#include "orte/util/hnp_contact.h"
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
#include "orte/mca/debugger/base/base.h"
Restore the ability to forcibly terminate by providing multiple ctrl-c's This commit was SVN r18618.
2008-06-09 17:08:54 +04:00
#include "orte/mca/odls/odls.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/mca/plm/plm.h"
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
#include "orte/mca/plm/base/plm_private.h"
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#include "orte/mca/ras/ras.h"
Bring in an updated launch system for the orteds. This commit restores the ability to execute singletons and singleton comm_spawn, both in single node and multi-node environments. Short description: major changes include - 1. singletons now fork/exec a local daemon to manage their operations. 2. the orte daemon code now resides in libopen-rte 3. daemons no longer use the orte triggering system during startup. Instead, they directly call back to their parent pls component to report ready to operate. A base function to count the callbacks has been provided. I have modified all the pls components except xcpu and poe (don't understand either well enough to do it). Full functionality has been verified for rsh, SLURM, and TM systems. Compile has been verified for xgrid and gridengine. This commit was SVN r15390.
2007-07-12 23:53:18 +04:00
#include "orte/mca/rml/rml.h"
- On the way to get the BTLs split out and lessen dependency on orte: Often, orte/util/show_help.h is included, although no functionality is required -- instead, most often opal_output.h, or orte/mca/rml/rml_types.h Please see orte_show_help_replacement.sh commited next. - Local compilation (Linux/x86_64) w/ -Wimplicit-function-declaration actually showed two *missing* #include "orte/util/show_help.h" in orte/mca/odls/base/odls_base_default_fns.c and in orte/tools/orte-top/orte-top.c Manually added these. Let's have MTT the last word. This commit was SVN r20557.
2009-02-14 05:26:12 +03:00
#include "orte/mca/rml/rml_types.h"
Cleanup and fix bugs in the MPI dynamics section. Modify the dpm API so it properly takes ports instead of process names (as correctly identified by Aurelien). Fix race conditions in the use of ompi-server. Fix incompatibilities between the mpi bindings and the dpm implemenation that could cause segfaults due to uninitialized memory. Fix the ompi-server -h cmd line option so it actually tells you something! Add two new testing codes to the orte/test/mpi area: accept and connect. This commit was SVN r18176.
2008-04-16 18:27:42 +04:00
#include "orte/mca/rml/base/rml_contact.h"
Add trace to the daemons and orterun so we can tell when their callbacks are being exercised. This commit was SVN r7432.
2005-09-19 21:20:01 +04:00
#include "orte/mca/errmgr/errmgr.h"
Add a sensor framework to ORTE that monitors applications and notifies the errmgr when they exceed specified boundaries. Two modules are included here: 1. file activity - can monitor file size, access and modification times. If these fail to change over a specified number of sampling iterations (rate is an mca param), then the errmgr is notified. 2. memory usage - checks amount of memory used by a process. Limit and sampling rate can be set. This support must be enabled by configuring --enable-sensors. ompi_info and orte-info have been updated to include the new framework. Also includes some initial steps toward restoring the recovery capability. Most notably, the ODLS API has been extended to include a "restart_proc" entry for restarting a local process, and organizes the various ERRMGR framework globals into a single struct as we do in the other ORTE frameworks. Fix an oversight in the ERRMGR framework where a pointer array was constructed, but not initialized. Implementation continues. This commit was SVN r23043.
2010-04-27 02:15:57 +04:00
#include "orte/mca/errmgr/base/errmgr_private.h"
Take the next step towards fully utilizing static ports for the daemons to eliminate the initial "phone home" to mpirun by modifying the orted termination procedure to eliminate the need for a full barrier-like operation. Instead, we add a "onesided" barrier to the grpcomm framework API that releases the orted once it has completed its own contribution to the barrier - i.e., the orteds now exit as the "ack" message rolls up towards mpirun instead of sending the "ack" directly to mpirun. This causes the orteds in the routing tree to remain alive until all termination "acks" from orteds below them have passed through. Thus, if we use static ports, we no longer require a direct orted-to-mpirun connection. Also modify the binomial routed module so it conforms to what all the other routed modules do and have all messages pass along the routing tree instead of short-circuiting between orteds. This further reduces the number of ports being opened on backend nodes. This commit was SVN r21203.
2009-05-11 18:11:44 +04:00
#include "orte/mca/grpcomm/grpcomm.h"
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Add trace to the daemons and orterun so we can tell when their callbacks are being exercised. This commit was SVN r7432.
2005-09-19 21:20:01 +04:00
#include "orte/runtime/runtime.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/orte_globals.h"
Add trace to the daemons and orterun so we can tell when their callbacks are being exercised. This commit was SVN r7432.
2005-09-19 21:20:01 +04:00
#include "orte/runtime/orte_wait.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/orte_data_server.h"
Afraid this has a couple of things mixed into the commit. Couldn't be helped - had missed one commit prior to running out the door on vacation. Fix race conditions in abnormal terminations. We had done a first-cut at this in a prior commit. However, the window remained partially open due to the fact that the HNP has multiple paths leading to orte_finalize. Most of our frameworks don't care if they are finalized more than once, but one of them does, which meant we segfaulted if orte_finalize got called more than once. Besides, we really shouldn't be doing that anyway. So we now introduce a set of atomic locks that prevent us from multiply calling abort, attempting to call orte_finalize, etc. My initial tests indicate this is working cleanly, but since it is a race condition issue, more testing will have to be done before we know for sure that this problem has been licked. Also, some updates relevant to the tool comm library snuck in here. Since those also touched the orted code (as did the prior changes), I didn't want to attempt to separate them out - besides, they are coming in soon anyway. More on them later as that functionality approaches completion. This commit was SVN r17843.
2008-03-17 20:58:59 +03:00
#include "orte/runtime/orte_locks.h"
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
#include "orte/runtime/orte_quit.h"
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Bring in an updated launch system for the orteds. This commit restores the ability to execute singletons and singleton comm_spawn, both in single node and multi-node environments. Short description: major changes include - 1. singletons now fork/exec a local daemon to manage their operations. 2. the orte daemon code now resides in libopen-rte 3. daemons no longer use the orte triggering system during startup. Instead, they directly call back to their parent pls component to report ready to operate. A base function to count the callbacks has been provided. I have modified all the pls components except xcpu and poe (don't understand either well enough to do it). Full functionality has been verified for rsh, SLURM, and TM systems. Compile has been verified for xgrid and gridengine. This commit was SVN r15390.
2007-07-12 23:53:18 +04:00
/* ensure I can behave like a daemon */
#include "orte/orted/orted.h"
A little more clean-up. TotalView now works with --enable-debug build. Tested with: pls = rsh totalview.6.6.0-2 Linux cadillac82.ccstar.lanl.gov 2.4.24 #1 SMP Thu Jul 1 15:28:04 MDT 2004 i686 i686 i386 GNU/Linux This commit was SVN r7108.
2005-08-31 20:15:59 +04:00
#include "orterun.h"
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/*
* Globals
*/
Initialize pointers to NULL This commit was SVN r23422.
2010-07-15 20:33:57 +04:00
static orte_job_t *jdata=NULL;
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
static char **global_mca_env = NULL;
With the branch to 1.2 made.... Clean up the remainder of the size_t references in the runtime itself. Convert to orte_std_cntr_t wherever it makes sense (only avoid those places where the actual memory size is referenced). Remove the obsolete oob barrier function (we actually obsoleted it a long time ago - just never bothered to clean it up). I have done my best to go through all the components and catch everything, even if I couldn't test compile them since I wasn't on that type of system. Still, I cannot guarantee that problems won't show up when you test this on specific systems. Usually, these will just show as "warning: comparison between signed and unsigned" notes which are easily fixed (just change a size_t to orte_std_cntr_t). In some places, people didn't use size_t, but instead used some other variant (e.g., I found several places with uint32_t). I tried to catch all of them, but... Once we get all the instances caught and fixed, this should once and for all resolve many of the heterogeneity problems. This commit was SVN r11204.
2006-08-15 23:54:10 +04:00
static orte_std_cntr_t total_num_apps = 0;
Fixes trac:377 Add --enable-orterun-prefix-by-default (and a synonym: --enable-mpirun-prefix-by-default) to make orterun always behave as if "--prefix $prefix" was given on the command line (where $prefix is the value given to the --prefix option to configure). This prevents many rsh/ssh users from needing to modify their shell startup files to set the LD_LIBRARY_PATH for Open MPI (they will still need to set PATH or otherwise find the OMPI executables to mpicc/mpirun/etc. their MPI applications). Also added --noprefix option to orterun to disable this behavior. Finally, note that even if --enable-orterun-prefix-by-default is specified, if the user specifies --prefix or /path/to/mpirun, these options will override the default value of the prefix ($prefix). This commit was SVN r11669. The following Trac tickets were found above: Ticket 377 --> https://svn.open-mpi.org/trac/ompi/ticket/377
2006-09-15 06:52:08 +04:00
static bool want_prefix_by_default = (bool) ORTE_WANT_ORTERUN_PREFIX_BY_DEFAULT;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
static char *ompi_server=NULL;
Fix two Coverity complaints CID 813 (value defined and not used) and 1039 (resource leak). While doing so, found and fixed another less obvious memory leak. This commit was SVN r18641.
2008-06-10 21:53:28 +04:00
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/*
Add support for DDT parallel debugger, which required several things: * Making some symbols and types be global (vs. static) in orterun * Adding a "ddt" entry in the MCA parameter orte_base_user_debugger default value * Add support for @executable@, @executable_argv@, and @single_app@ tokens in the orte_base_user_debugger MCA parameter. * Added various error checks and corresponding help messages after finding a debugger in the PATH Fixes trac:1081 This commit was SVN r15323. The following Trac tickets were found above: Ticket 1081 --> https://svn.open-mpi.org/trac/ompi/ticket/1081
2007-07-10 16:53:48 +04:00
* Globals
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
*/
Fix a race condition - ensure we don't call terminate in orterun more than once, even if the timeout fires while we are doing so This commit was SVN r17766.
2008-03-06 22:35:57 +03:00
struct orterun_globals_t orterun_globals;
static bool globals_init = false;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Fix a race condition - ensure we don't call terminate in orterun more than once, even if the timeout fires while we are doing so This commit was SVN r17766.
2008-03-06 22:35:57 +03:00
static opal_cmd_line_init_t cmd_line_init[] = {
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Various "obvious" options */
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
{ NULL, NULL, NULL, 'h', NULL, "help", 0,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
&orterun_globals.help, OPAL_CMD_LINE_TYPE_BOOL,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"This help message" },
Fix for ticket #91 mpirun/orterun now has an option to print the version number. If -V/--version is given, it will print the version number. If it's the only option, we exit cleanly. Otherwise, we continue on as if --version wasn't given (except we've printed the version number). --This line, and th se below, will be ignored-- M orte/tools/orterun/orterun.c M orte/tools/orterun/help-orterun.txt This commit was SVN r10276.
2006-06-09 21:21:23 +04:00
{ NULL, NULL, NULL, 'V', NULL, "version", 0,
&orterun_globals.version, OPAL_CMD_LINE_TYPE_BOOL,
"Print version and exit" },
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
{ NULL, NULL, NULL, 'v', NULL, "verbose", 0,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
&orterun_globals.verbose, OPAL_CMD_LINE_TYPE_BOOL,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"Be verbose" },
Truly respect the -quiet option. Make it an mca param so someone doesn't have to put it solely on the cmd line. Tell show_help to shaddup as well. This commit was SVN r22926.
2010-04-02 18:19:38 +04:00
{ "orte", "execute", "quiet", 'q', NULL, "quiet", 0,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
* add --quiet option, and surpress a couple of the status messages in orterun if it is actually enabled. For ticket #129. This commit was SVN r10497.
2006-06-26 22:21:45 +04:00
"Suppress helpful messages" },
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
{ NULL, NULL, NULL, '\0', "report-pid", "report-pid", 1,
&orterun_globals.report_pid, OPAL_CMD_LINE_TYPE_STRING,
"Printout pid on stdout [-], stderr [+], or a file [anything else]" },
{ NULL, NULL, NULL, '\0', "report-uri", "report-uri", 1,
&orterun_globals.report_uri, OPAL_CMD_LINE_TYPE_STRING,
"Printout URI on stdout [-], stderr [+], or a file [anything else]" },
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
/* exit status reporting */
{ "orte", "report", "child_jobs_separately", '\0', "report-child-jobs-separately", "report-child-jobs-separately", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Return the exit status of the primary job only" },
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
/* hetero apps */
Minimize the amount of topology info returned by the daemons. Most clusters, especially at scale, use the same node topology on every node, so there is no re ason to return the topology from every daemon. Borrow a page from the --hetero-apps page and let users indicate that the node topology differs by adding a -- hetero-nodes option to mpirun. If the option is set, then every daemon returns topology info. If not set, then only daemon vpid=1 returns it. We always want one daemon to return the topology as the head node is often different from the compute nodes. Having one daemon return the compute node topolo gy allows us to detect any such difference. All compute nodes are then set to the same topology. This commit was SVN r25408.
2011-11-01 22:43:10 +04:00
{ "orte", "hetero", "apps", '\0', NULL, "hetero-apps", 0,
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Indicates that multiple app_contexts are being provided that are a mix of 32/64 bit binaries" },
Add the -xml option to mpirun to indicate that xml output is desired This commit was SVN r18539.
2008-05-29 18:11:31 +04:00
/* select XML output */
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
{ "orte", "xml", "output", '\0', "xml", "xml", 0,
Initial work on xml support - allocation and job map outputs completed. More to come. This commit was SVN r18587.
2008-06-05 00:53:12 +04:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
Add the -xml option to mpirun to indicate that xml output is desired This commit was SVN r18539.
2008-05-29 18:11:31 +04:00
"Provide all output in XML format" },
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
{ "orte", "xml", "file", '\0', "xml-file", "xml-file", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Provide all output in XML format to the specified file" },
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
/* tag output */
{ "orte", "tag", "output", '\0', "tag-output", "tag-output", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Tag all output with [job,rank]" },
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
{ "orte", "timestamp", "output", '\0', "timestamp-output", "timestamp-output", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Timestamp all application process output" },
{ "orte", "output", "filename", '\0', "output-filename", "output-filename", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Redirect output from application processes into filename.rank" },
{ "orte", "xterm", NULL, '\0', "xterm", "xterm", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Create a new xterm window and display output from the specified ranks there" },
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
/* select stdin option */
{ NULL, NULL, NULL, '\0', "stdin", "stdin", 1,
&orterun_globals.stdin_target, OPAL_CMD_LINE_TYPE_STRING,
"Specify procs to receive stdin [rank, all, none] (default: 0, indicating rank 0)" },
Per the July technical meeting: Standardize the handling of the orte launch agent option across PLMs. This has been a consistent complaint I have received - each PLM would register its own MCA param to get input on the launch agent for remote nodes (in fact, one or two didn't, but most did). This would then get handled in various and contradictory ways. Some PLMs would accept only a one-word input. Others accepted multi-word args such as "valgrind orted", but then some would error by putting any prefix specified on the cmd line in front of the incorrect argument. For example, while using the rsh launcher, if you specified "valgrind orted" as your launch agent and had "--prefix foo" on you cmd line, you would attempt to execute "ssh foo/valgrind orted" - which obviously wouldn't work. This was all -very- confusing to users, who had to know which PLM was being used so they could even set the right mca param in the first place! And since we don't warn about non-recognized or non-used mca params, half of the time they would wind up not doing what they thought they were telling us to do. To solve this problem, we did the following: 1. removed all mca params from the individual plms for the launch agent 2. added a new mca param "orte_launch_agent" for this purpose. To further simplify for users, this comes with a new cmd line option "--launch-agent" that can take a multi-word string argument. The value of the param defaults to "orted". 3. added a PLM base function that processes the orte_launch_agent value and adds the contents to a provided argv array. This can subsequently be harvested at-will to handle multi-word values 4. modified the PLMs to use this new function. All the PLMs except for the rsh PLM required very minor change - just called the function and moved on. The rsh PLM required much larger changes as - because of the rsh/ssh cmd line limitations - we had to correctly prepend any provided prefix to the correct argv entry. 5. added a new opal_argv_join_range function that allows the caller to "join" argv entries between two specified indices Please let me know of any problems. I tried to make this as clean as possible, but cannot compile all PLMs to ensure all is correct. This commit was SVN r19097.
2008-07-30 22:26:24 +04:00
/* Specify the launch agent to be used */
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
{ "orte", "launch", "agent", '\0', "launch-agent", "launch-agent", 1,
Per the July technical meeting: Standardize the handling of the orte launch agent option across PLMs. This has been a consistent complaint I have received - each PLM would register its own MCA param to get input on the launch agent for remote nodes (in fact, one or two didn't, but most did). This would then get handled in various and contradictory ways. Some PLMs would accept only a one-word input. Others accepted multi-word args such as "valgrind orted", but then some would error by putting any prefix specified on the cmd line in front of the incorrect argument. For example, while using the rsh launcher, if you specified "valgrind orted" as your launch agent and had "--prefix foo" on you cmd line, you would attempt to execute "ssh foo/valgrind orted" - which obviously wouldn't work. This was all -very- confusing to users, who had to know which PLM was being used so they could even set the right mca param in the first place! And since we don't warn about non-recognized or non-used mca params, half of the time they would wind up not doing what they thought they were telling us to do. To solve this problem, we did the following: 1. removed all mca params from the individual plms for the launch agent 2. added a new mca param "orte_launch_agent" for this purpose. To further simplify for users, this comes with a new cmd line option "--launch-agent" that can take a multi-word string argument. The value of the param defaults to "orted". 3. added a PLM base function that processes the orte_launch_agent value and adds the contents to a provided argv array. This can subsequently be harvested at-will to handle multi-word values 4. modified the PLMs to use this new function. All the PLMs except for the rsh PLM required very minor change - just called the function and moved on. The rsh PLM required much larger changes as - because of the rsh/ssh cmd line limitations - we had to correctly prepend any provided prefix to the correct argv entry. 5. added a new opal_argv_join_range function that allows the caller to "join" argv entries between two specified indices Please let me know of any problems. I tried to make this as clean as possible, but cannot compile all PLMs to ensure all is correct. This commit was SVN r19097.
2008-07-30 22:26:24 +04:00
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Command used to start processes on remote nodes (default: orted)" },
Merging in the jjhursey-ft-cr-stable branch (r13912 : HEAD). This merge adds Checkpoint/Restart support to Open MPI. The initial frameworks and components support a LAM/MPI-like implementation. This commit follows the risk assessment presented to the Open MPI core development group on Feb. 22, 2007. This commit closes trac:158 More details to follow. This commit was SVN r14051. The following SVN revisions from the original message are invalid or inconsistent and therefore were not cross-referenced: r13912 The following Trac tickets were found above: Ticket 158 --> https://svn.open-mpi.org/trac/ompi/ticket/158
2007-03-17 02:11:45 +03:00
/* Preload the binary on the remote machine */
{ NULL, NULL, NULL, 's', NULL, "preload-binary", 0,
&orterun_globals.preload_binary, OPAL_CMD_LINE_TYPE_BOOL,
"Preload the binary on the remote machine before starting the remote process." },
/* Preload files on the remote machine */
{ NULL, NULL, NULL, '\0', NULL, "preload-files", 1,
&orterun_globals.preload_files, OPAL_CMD_LINE_TYPE_STRING,
"Preload the comma separated list of files to the remote machines current working directory before starting the remote process." },
/* Where to Preload files on the remote machine */
{ NULL, NULL, NULL, '\0', NULL, "preload-files-dest-dir", 1,
&orterun_globals.preload_files_dest_dir, OPAL_CMD_LINE_TYPE_STRING,
"The destination directory to use in conjunction with --preload-files. By default the absolute and relative paths provided by --preload-files are used." },
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_FT_CR == 1
/* Tell SStore to preload a snapshot before launch */
{ NULL, NULL, NULL, '\0', NULL, "sstore-load", 1,
&orterun_globals.sstore_load, OPAL_CMD_LINE_TYPE_STRING,
"Internal Use Only! Tell SStore to preload a snapshot before launch." },
#endif
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Use an appfile */
{ NULL, NULL, NULL, '\0', NULL, "app", 1,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
&orterun_globals.appfile, OPAL_CMD_LINE_TYPE_STRING,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"Provide an appfile; ignore all other command line options" },
/* Number of processes; -c, -n, --n, -np, and --np are all
synonyms */
{ NULL, NULL, NULL, 'c', "np", "np", 1,
Fix for #397 on 64 bit platforms sizeof(size_t) != sizeof(orte_std_cntr_t), and we were incorrectly assuming this when dealing with num procs. It worked on little endian platforms, but not big endian. So change num_procs to type int, and cast where needed. This commit was SVN r11796.
2006-09-25 23:41:54 +04:00
&orterun_globals.num_procs, OPAL_CMD_LINE_TYPE_INT,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"Number of processes to run" },
{ NULL, NULL, NULL, '\0', "n", "n", 1,
Fix for #397 on 64 bit platforms sizeof(size_t) != sizeof(orte_std_cntr_t), and we were incorrectly assuming this when dealing with num procs. It worked on little endian platforms, but not big endian. So change num_procs to type int, and cast where needed. This commit was SVN r11796.
2006-09-25 23:41:54 +04:00
&orterun_globals.num_procs, OPAL_CMD_LINE_TYPE_INT,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"Number of processes to run" },
Per LANL's stated need, add functionality that runs a.out across ALL available process slots if no num_proc is specified on the command line. However, please note the following limitation: we ONLY allow ONE application to be specified on the command line when this feature is invoked. If multiple apps are specified, the user MUST also specify the number to be launched for each and every one of them. Update the help text to report errors when not following that rule. Also updated the RMAPS help text to reflect the reorganization of some of the round-robin code into the base. The new functionality has been tested under Mac OS-X and on Odin using an MPI program. Both byslot and bynode mapping have been checked and verified. Operational support for other systems needs to be verified - I respectfully request people's help in doing so. This commit was SVN r10708.
2006-07-11 01:25:33 +04:00
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Set a hostfile */
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
{ NULL, NULL, NULL, '\0', "hostfile", "hostfile", 1,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
NULL, OPAL_CMD_LINE_TYPE_STRING,
* sync with close to the trunk, but right before the gm changes that we actually need. More, interesting, stuff coming soon. This commit was SVN r4933.
2005-03-19 02:40:08 +03:00
"Provide a hostfile" },
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
{ NULL, NULL, NULL, '\0', "machinefile", "machinefile", 1,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
NULL, OPAL_CMD_LINE_TYPE_STRING,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"Provide a hostfile" },
Implement the seq rmaps module that sequentially maps process ranks to a list hosts in a hostfile. Restore the "do-not-launch" functionality so users can test a mapping without launching it. Add a "do-not-resolve" cmd line flag to mpirun so the opal/util/if.c code does not attempt to resolve network addresses, thus enabling a user to test a hostfile mapping without hanging on network resolve requests. Add a function to hostfile to generate an ordered list of host names from a hostfile This commit was SVN r18190.
2008-04-17 17:50:59 +04:00
{ "orte", "default", "hostfile", '\0', "default-hostfile", "default-hostfile", 1,
Add default hostfile parameter plus --default-hostfile command line option. Fix error message when job setup failed This commit was SVN r17724.
2008-03-05 07:54:57 +03:00
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Provide a default hostfile" },
Implement the seq rmaps module that sequentially maps process ranks to a list hosts in a hostfile. Restore the "do-not-launch" functionality so users can test a mapping without launching it. Add a "do-not-resolve" cmd line flag to mpirun so the opal/util/if.c code does not attempt to resolve network addresses, thus enabling a user to test a hostfile mapping without hanging on network resolve requests. Add a function to hostfile to generate an ordered list of host names from a hostfile This commit was SVN r18190.
2008-04-17 17:50:59 +04:00
{ "opal", "if", "do_not_resolve", '\0', "do-not-resolve", "do-not-resolve", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Do not attempt to resolve interfaces" },
Add default hostfile parameter plus --default-hostfile command line option. Fix error message when job setup failed This commit was SVN r17724.
2008-03-05 07:54:57 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* uri of Open MPI server, or at least where to get it */
{ NULL, NULL, NULL, '\0', "ompi-server", "ompi-server", 1,
&orterun_globals.ompi_server, OPAL_CMD_LINE_TYPE_STRING,
Fix ompi-server so it works with unity routed module - still not working with tree routing. Cleanup debug flag so it activates debugging on the data server code itself This commit was SVN r18080.
2008-04-04 23:17:28 +04:00
"Specify the URI of the Open MPI server, or the name of the file (specified as file:filename) that contains that info" },
Per the July technical meeting: During the discussion of MPI-2 functionality, it was pointed out by Aurelien that there was an inherent race condition between startup of ompi-server and mpirun. Specifically, if someone started ompi-server to run in the background as part of a script, and then immediately executed mpirun, it was possible that an MPI proc could attempt to contact the server (or that mpirun could try to read the server's contact file before the server is running and ready. At that time, we discussed createing a new tool "ompi-wait-server" that would wait for the server to be running, and/or probe to see if it is running and return true/false. However, rather than create yet another tool, it seemed just as effective to add the functionality to mpirun. Thus, this commit creates two new mpirun cmd line flags (hey, you can never have too many!): --wait-for-server : instructs mpirun to ping the server to see if it responds. This causes mpirun to execute an rml.ping to the server's URI with an appropriate timeout interval - if the ping isn't successful, mpirun attempts it again. --server-wait-time xx : sets the ping timeout interval to xx seconds. Note that mpirun will attempt to ping the server twice with this timeout, so we actually wait for twice this time. Default is 10 seconds, which should be plenty of time. This has only lightly been tested. It works if the server is present, and outputs a nice error message if it cannot be contacted. I have not tested the race condition case. This commit was SVN r19152.
2008-08-05 00:29:50 +04:00
{ NULL, NULL, NULL, '\0', "wait-for-server", "wait-for-server", 0,
&orterun_globals.wait_for_server, OPAL_CMD_LINE_TYPE_BOOL,
"If ompi-server is not already running, wait until it is detected (default: false)" },
{ NULL, NULL, NULL, '\0', "server-wait-time", "server-wait-time", 1,
&orterun_globals.server_wait_timeout, OPAL_CMD_LINE_TYPE_INT,
"Time in seconds to wait for ompi-server (default: 10 sec)" },
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Move the carto framework to the trunk. This commit was SVN r17177.
2008-01-23 12:20:34 +03:00
{ "carto", "file", "path", '\0', "cf", "cartofile", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Provide a cartography file" },
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Correct a couple of bugs in the rank_file mapper that were incorrectly assigning vpids. Add a capability to parse the rankfile to extract node information in place of requiring both hostfile and rankfile for non-RM managed environments. The rankfile is -only- parsed for this IF the hostfile and -host options are not given. Otherwise, those are used to establish allocation info as we did before this commit. This commit was SVN r21815.
2009-08-13 20:08:43 +04:00
{ "orte", "rankfile", NULL, '\0', "rf", "rankfile", 1,
Added -rf|--rankfile <arg0> oprion to mpirun -h + man info This commit was SVN r18812.
2008-07-07 17:46:22 +04:00
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Provide a rankfile file" },
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Export environment variables; potentially used multiple times,
so it does not make sense to set into a variable */
{ NULL, NULL, NULL, 'x', NULL, NULL, 1,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
NULL, OPAL_CMD_LINE_TYPE_NULL,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"Export an environment variable, optionally specifying a value (e.g., \"-x foo\" exports the environment variable foo and takes its value from the current environment; \"-x foo=bar\" exports the environment variable name foo and sets its value to \"bar\" in the started processes)" },
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* Mapping controls */
Modify the "--display-map-at-launch" option to just "--display-map". Now that we have a "--do-not-launch" option, the "-at-launch" part of the display-map option was confusing. "--display-map" displays the resulting process map before we launch anyway, so this is clearer. This commit was SVN r12840.
2006-12-13 16:49:15 +03:00
{ "rmaps", "base", "display_map", '\0', "display-map", "display-map", 0,
- Silence compiler on ORTE_TRANSPORT_KEY_FMT, it is fixed to llx - No functional changes, just indentation and corrections to error output. This commit was SVN r12734.
2006-12-03 16:59:23 +03:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Display the process map just before launch"},
Modify the --display-map and --display-alloc per note on devel list to reduce info for user understanding. Add --display-devel-map and --display-devel-alloc to display all the detailed info we used to provide - it is only of use/interest to developers anyway and confuses users. This commit was SVN r19608.
2008-09-23 19:46:34 +04:00
{ "rmaps", "base", "display_devel_map", '\0', "display-devel-map", "display-devel-map", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Display a detailed process map (mostly intended for developers) just before launch"},
Add a simple pattern mapper as an example of how to use the topology info to create desired mappings. Let the user specify a pattern based on resource types, and map that pattern across all available nodes as resources permit. Don't automatically display the topology for each node when --display-devel-map is set as it can overwhelm the reader. Use a separate flag --display-topo to get it. This commit was SVN r25396.
2011-10-29 19:12:45 +04:00
{ "rmaps", "base", "display_topo_with_map", '\0', "display-topo", "display-topo", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Display the topology as part of the process map (mostly intended for developers) just before launch"},
Add an option for printing a diffable process map for testing mappers This commit was SVN r25428.
2011-11-03 18:22:07 +04:00
{ "rmaps", "base", "display_diffable_map", '\0', "display-diffable-map", "display-diffable-map", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Display a diffable process map (mostly intended for developers) just before launch"},
Add the -xml option to mpirun to indicate that xml output is desired This commit was SVN r18539.
2008-05-29 18:11:31 +04:00
{ NULL, NULL, NULL, 'H', "host", "host", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"List of hosts to invoke processes on" },
{ "rmaps", "base", "no_schedule_local", '\0', "nolocal", "nolocal", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Do not run any MPI applications on the local node" },
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
{ "rmaps", "base", "no_oversubscribe", '\0', "nooversubscribe", "nooversubscribe", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Nodes are not to be oversubscribed, even if the system supports such operation"},
{ "rmaps", "base", "oversubscribe", '\0', "oversubscribe", "oversubscribe", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Nodes are allowed to be oversubscribed, even on a managed system"},
#if 0
Setup cpus-per-proc and cpus-per-rank as synonyms, both in mca params and on mpirun cmd line This commit was SVN r21914.
2009-08-30 18:30:36 +04:00
{ "rmaps", "base", "cpus_per_rank", '\0', "cpus-per-proc", "cpus-per-proc", 1,
NULL, OPAL_CMD_LINE_TYPE_INT,
"Number of cpus to use for each process [default=1]" },
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
{ "rmaps", "base", "cpus_per_rank", '\0', "cpus-per-rank", "cpus-per-rank", 1,
NULL, OPAL_CMD_LINE_TYPE_INT,
Setup cpus-per-proc and cpus-per-rank as synonyms, both in mca params and on mpirun cmd line This commit was SVN r21914.
2009-08-30 18:30:36 +04:00
"Synonym for cpus-per-proc" },
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#endif
/* backward compatiblity */
{ "rmaps", "base", "bynode", '\0', "bynode", "bynode", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Whether to map and rank processes round-robin by node" },
{ "rmaps", "base", "byslot", '\0', "byslot", "byslot", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Whether to map and rank processes round-robin by slot" },
/* Nperxxx options that do not require topology and are always
* available - included for backwards compatibility
*/
{ "rmaps", "ppr", "pernode", '\0', "pernode", "pernode", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Launch one process per available node" },
{ "rmaps", "ppr", "n_pernode", '\0', "npernode", "npernode", 1,
NULL, OPAL_CMD_LINE_TYPE_INT,
"Launch n processes per node on all allocated nodes" },
#if OPAL_HAVE_HWLOC
/* declare hardware threads as independent cpus */
{ "hwloc", "base", "use_hwthreads_as_cpus", '\0', "use-hwthread-cpus", "use-hwthread-cpus", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Use hardware threads as independent cpus" },
/* include npersocket for backwards compatibility */
{ "rmaps", "ppr", "n_persocket", '\0', "npersocket", "npersocket", 1,
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
NULL, OPAL_CMD_LINE_TYPE_INT,
"Launch n processes per socket on all allocated nodes" },
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* Mapping options */
{ "rmaps", "base", "mapping_policy", '\0', NULL, "map-by", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Mapping Policy [slot (default) | hwthread | core | socket | numa | board | node]" },
/* Ranking options */
{ "rmaps", "base", "ranking_policy", '\0', NULL, "rank-by", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Ranking Policy [slot (default) | hwthread | core | socket | numa | board | node]" },
/* Binding options */
{ "hwloc", "base", "binding_policy", '\0', NULL, "bind-to", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Policy for binding processes [none (default) | hwthread | core | socket | numa | board] (supported qualifiers: overload-allowed,if-supported)" },
/* backward compatiblity */
{ "hwloc", "base", "bind_to_core", '\0', "bind-to-core", "bind-to-core", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Bind processes to cores" },
{ "hwloc", "base", "bind_to_socket", '\0', "bind-to-socket", "bind-to-socket", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Bind processes to sockets" },
{ "hwloc", "base", "report_bindings", '\0', "report-bindings", "report-bindings", 0,
Add a --report-bindings cmd line option to mpirun This commit was SVN r21829.
2009-08-18 21:10:23 +04:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
Instead of an odls-base mca param, make report_bindings a global param so that we can (a) detect it was set in the plm, and then (b) ensure it gets passed along to remote orteds so they will comply with the request. This commit was SVN r22021.
2009-09-28 07:17:15 +04:00
"Whether to report process bindings to stderr" },
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* slot list option */
{ "hwloc", "base", "slot_list", '\0', "slot-list", "slot-list", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"List of processor IDs to bind processes to [default=NULL]"},
/* generalized pattern mapping option */
{ "rmaps", "ppr", "pattern", '\0', NULL, "ppr", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Comma-separated list of number of processes on a given resource type [default: none]" },
#else
/* Mapping options */
{ "rmaps", "base", "mapping_policy", '\0', NULL, "map-by", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Mapping Policy [slot (default) | node]" },
/* Ranking options */
{ "rmaps", "base", "ranking_policy", '\0', NULL, "rank-by", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Ranking Policy [slot (default) | node]" },
#endif
Add the -xml option to mpirun to indicate that xml output is desired This commit was SVN r18539.
2008-05-29 18:11:31 +04:00
/* Allocation options */
Add --display-allocation option to orterun that will display the node-by-node information regarding your allocation. This commit was SVN r18216.
2008-04-20 06:25:45 +04:00
{ "ras", "base", "display_alloc", '\0', "display-allocation", "display-allocation", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Display the allocation being used by this job"},
Modify the --display-map and --display-alloc per note on devel list to reduce info for user understanding. Add --display-devel-map and --display-devel-alloc to display all the detailed info we used to provide - it is only of use/interest to developers anyway and confuses users. This commit was SVN r19608.
2008-09-23 19:46:34 +04:00
{ "ras", "base", "display_devel_alloc", '\0', "display-devel-allocation", "display-devel-allocation", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Display a detailed list (mostly intended for developers) of the allocation being used by this job"},
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#if OPAL_HAVE_HWLOC
{ "hwloc", "base", "cpu_set", '\0', "cpu-set", "cpu-set", 1,
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Comma-separated list of ranges specifying logical cpus allocated to this job [default: none]"},
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#endif
{ NULL, NULL, NULL, 'H', "host", "host", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"List of hosts to invoke processes on" },
Add --display-allocation option to orterun that will display the node-by-node information regarding your allocation. This commit was SVN r18216.
2008-04-20 06:25:45 +04:00
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* mpiexec-like arguments */
{ NULL, NULL, NULL, '\0', "wdir", "wdir", 1,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
&orterun_globals.wdir, OPAL_CMD_LINE_TYPE_STRING,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"Set the working directory of the started processes" },
Add -wdir option as a synonym for -wd (to make us match the man page). This commit was SVN r14614.
2007-05-08 23:09:32 +04:00
{ NULL, NULL, NULL, '\0', "wd", "wd", 1,
&orterun_globals.wdir, OPAL_CMD_LINE_TYPE_STRING,
"Synonym for --wdir" },
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
{ NULL, NULL, NULL, '\0', "path", "path", 1,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
&orterun_globals.path, OPAL_CMD_LINE_TYPE_STRING,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
"PATH to be used to look for executables to start processes" },
Final merge of stuff from /tmp/tm-stuff tree (merged through /tmp/tm-merge). Validated by RHC. Summary: - Add --nolocal (and -nolocal) options to orterun - Make some scalability improvements to the tm pls This commit was SVN r10651.
2006-07-05 00:12:35 +04:00
Good weekend brainless activity -- implement the orterun command line debugger scheme described in http://www.open-mpi.org/community/lists/users/2005/10/0214.php. This makes our user-level debugger scheme much more vendor-independent (although the "-tv" option will still work for backwards compatibility -- it'll just be a synonum of "--debug"). This commit was SVN r8206.
2005-11-20 19:06:53 +03:00
/* User-level debugger arguments */
{ NULL, NULL, NULL, '\0', "tv", "tv", 0,
&orterun_globals.debugger, OPAL_CMD_LINE_TYPE_BOOL,
"Deprecated backwards compatibility flag; synonym for \"--debug\"" },
{ NULL, NULL, NULL, '\0', "debug", "debug", 0,
&orterun_globals.debugger, OPAL_CMD_LINE_TYPE_BOOL,
"Invoke the user-level debugger indicated by the orte_base_user_debugger MCA parameter" },
{ "orte", "base", "user_debugger", '\0', "debugger", "debugger", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Sequence of debuggers to search for when \"--debug\" is used" },
Add a new MCA param (and corresponding mpirun cmd line option) to output the debugger proctable info after launch. The output is just the job map with the process pid included, so you get a node-by-node list of the process ranks on that node and thier pids. Works for initial launch and comm_spawn. xml and non-xml output is available This commit was SVN r22725.
2010-02-27 11:32:25 +03:00
{ "orte", "output", "debugger_proctable", '\0', "output-proctable", "output-proctable", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Output the debugger proctable after launch" },
Bring in the remote launch changes. This still isn't fully functional, but impacted a few other places that were worth fixing. 1. Added a new function to launch head node processes on remote nodes. 2. Added new tool "orteprobe" that checks to see if a daemon is running on a node. If so, it reports the contact info back to the requestor. If not, it will (eventually - but not now) fork/exec a daemon on the node, report the contact info back to requestor, and then die. 3. Modified orted to handle universe name parameters, and added separate command line flags for debugging the daemon and saving daemon debugging output in a file. The "debug" flag now turns on the runtime debug info instead of the daemon debug - thus, you can now just get daemon debug info if you like. 4. Fix the dps to handle zero length strings correctly. 5. Modify the fork and rsh launchers to pass required environmental variables to the daemons and processes 6. Pulled the redirection of stdin/stdout/stderr for the daemon out of orted and put it into the daemon_init function to simplify orted logic. 7. Modified sys_info to correctly deal with passed mca param 8. Modified univ_info to parse incoming universe location information. This commit was SVN r5705.
2005-05-13 01:44:23 +04:00
/* OpenRTE arguments */
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
{ "orte", "debug", NULL, 'd', "debug-devel", "debug-devel", 0,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
Bring in the remote launch changes. This still isn't fully functional, but impacted a few other places that were worth fixing. 1. Added a new function to launch head node processes on remote nodes. 2. Added new tool "orteprobe" that checks to see if a daemon is running on a node. If so, it reports the contact info back to the requestor. If not, it will (eventually - but not now) fork/exec a daemon on the node, report the contact info back to requestor, and then die. 3. Modified orted to handle universe name parameters, and added separate command line flags for debugging the daemon and saving daemon debugging output in a file. The "debug" flag now turns on the runtime debug info instead of the daemon debug - thus, you can now just get daemon debug info if you like. 4. Fix the dps to handle zero length strings correctly. 5. Modify the fork and rsh launchers to pass required environmental variables to the daemons and processes 6. Pulled the redirection of stdin/stdout/stderr for the daemon out of orted and put it into the daemon_init function to simplify orted logic. 7. Modified sys_info to correctly deal with passed mca param 8. Modified univ_info to parse incoming universe location information. This commit was SVN r5705.
2005-05-13 01:44:23 +04:00
"Enable debugging of OpenRTE" },
Add a couple of options to orterun that support debugging of daemons for memory corruption. Ensure that the environment provided to local application processes isn't "polluted" by the orteds This commit was SVN r12087.
2006-10-11 19:18:57 +04:00
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
{ "orte", "debug", "daemons", '\0', "debug-daemons", "debug-daemons", 0,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
NULL, OPAL_CMD_LINE_TYPE_INT,
Bring in the remote launch changes. This still isn't fully functional, but impacted a few other places that were worth fixing. 1. Added a new function to launch head node processes on remote nodes. 2. Added new tool "orteprobe" that checks to see if a daemon is running on a node. If so, it reports the contact info back to the requestor. If not, it will (eventually - but not now) fork/exec a daemon on the node, report the contact info back to requestor, and then die. 3. Modified orted to handle universe name parameters, and added separate command line flags for debugging the daemon and saving daemon debugging output in a file. The "debug" flag now turns on the runtime debug info instead of the daemon debug - thus, you can now just get daemon debug info if you like. 4. Fix the dps to handle zero length strings correctly. 5. Modify the fork and rsh launchers to pass required environmental variables to the daemons and processes 6. Pulled the redirection of stdin/stdout/stderr for the daemon out of orted and put it into the daemon_init function to simplify orted logic. 7. Modified sys_info to correctly deal with passed mca param 8. Modified univ_info to parse incoming universe location information. This commit was SVN r5705.
2005-05-13 01:44:23 +04:00
"Enable debugging of any OpenRTE daemons used by this application" },
Add a couple of options to orterun that support debugging of daemons for memory corruption. Ensure that the environment provided to local application processes isn't "polluted" by the orteds This commit was SVN r12087.
2006-10-11 19:18:57 +04:00
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
{ "orte", "debug", "daemons_file", '\0', "debug-daemons-file", "debug-daemons-file", 0,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
Bring in the remote launch changes. This still isn't fully functional, but impacted a few other places that were worth fixing. 1. Added a new function to launch head node processes on remote nodes. 2. Added new tool "orteprobe" that checks to see if a daemon is running on a node. If so, it reports the contact info back to the requestor. If not, it will (eventually - but not now) fork/exec a daemon on the node, report the contact info back to requestor, and then die. 3. Modified orted to handle universe name parameters, and added separate command line flags for debugging the daemon and saving daemon debugging output in a file. The "debug" flag now turns on the runtime debug info instead of the daemon debug - thus, you can now just get daemon debug info if you like. 4. Fix the dps to handle zero length strings correctly. 5. Modify the fork and rsh launchers to pass required environmental variables to the daemons and processes 6. Pulled the redirection of stdin/stdout/stderr for the daemon out of orted and put it into the daemon_init function to simplify orted logic. 7. Modified sys_info to correctly deal with passed mca param 8. Modified univ_info to parse incoming universe location information. This commit was SVN r5705.
2005-05-13 01:44:23 +04:00
"Enable debugging of any OpenRTE daemons used by this application, storing output in files" },
Add a couple of options to orterun that support debugging of daemons for memory corruption. Ensure that the environment provided to local application processes isn't "polluted" by the orteds This commit was SVN r12087.
2006-10-11 19:18:57 +04:00
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
{ "orte", "leave", "session_attached", '\0', "leave-session-attached", "leave-session-attached", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Enable debugging of OpenRTE" },
{ NULL, NULL, NULL, '\0', "tmpdir", "tmpdir", 1,
- Revert r20739 This commit was SVN r20742. The following SVN revision numbers were found above: r20739 --> open-mpi/ompi@781caee0b670ea169baead8facc23390112f094f
2009-03-06 00:56:03 +03:00
&orte_process_info.tmpdir_base, OPAL_CMD_LINE_TYPE_STRING,
Bring in the remote launch changes. This still isn't fully functional, but impacted a few other places that were worth fixing. 1. Added a new function to launch head node processes on remote nodes. 2. Added new tool "orteprobe" that checks to see if a daemon is running on a node. If so, it reports the contact info back to the requestor. If not, it will (eventually - but not now) fork/exec a daemon on the node, report the contact info back to requestor, and then die. 3. Modified orted to handle universe name parameters, and added separate command line flags for debugging the daemon and saving daemon debugging output in a file. The "debug" flag now turns on the runtime debug info instead of the daemon debug - thus, you can now just get daemon debug info if you like. 4. Fix the dps to handle zero length strings correctly. 5. Modify the fork and rsh launchers to pass required environmental variables to the daemons and processes 6. Pulled the redirection of stdin/stdout/stderr for the daemon out of orted and put it into the daemon_init function to simplify orted logic. 7. Modified sys_info to correctly deal with passed mca param 8. Modified univ_info to parse incoming universe location information. This commit was SVN r5705.
2005-05-13 01:44:23 +04:00
"Set the root for the session directory tree for orterun ONLY" },
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
{ "orte", "do_not", "launch", '\0', "do-not-launch", "do-not-launch", 0,
Implement the seq rmaps module that sequentially maps process ranks to a list hosts in a hostfile. Restore the "do-not-launch" functionality so users can test a mapping without launching it. Add a "do-not-resolve" cmd line flag to mpirun so the opal/util/if.c code does not attempt to resolve network addresses, thus enabling a user to test a hostfile mapping without hanging on network resolve requests. Add a function to hostfile to generate an ordered list of host names from a hostfile This commit was SVN r18190.
2008-04-17 17:50:59 +04:00
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Perform all necessary operations to prepare to launch the application, but do not actually launch it" },
Add a new option to orterun: "--do-not-launch" directs the system to do the allocation, map, job setup, etc., but don't actually launch the job. This lets us test all the setup portions of the code. Also, take the first step in updating how we handle mca params in ORTE - bring it closer to how it is done in the other two layers. Much more work to be done here. This commit was SVN r12838.
2006-12-13 07:51:38 +03:00
- Fix bug in orterun where --prefix didn't show up in the help output (reported by Cisco) - While in orterun, add a feature that multiple users have asked for: if you specify an absolute pathname to orterun, such as "/path/to/bin/orterun ...", it's equivalent to "orterun --path /path/to ..." This commit was SVN r9181.
2006-02-28 14:52:12 +03:00
{ NULL, NULL, NULL, '\0', NULL, "prefix", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Prefix where Open MPI is installed on remote nodes" },
Oops: --noprefix should not take an argument. This commit was SVN r12043.
2006-10-06 17:02:56 +04:00
{ NULL, NULL, NULL, '\0', NULL, "noprefix", 0,
Fixes trac:377 Add --enable-orterun-prefix-by-default (and a synonym: --enable-mpirun-prefix-by-default) to make orterun always behave as if "--prefix $prefix" was given on the command line (where $prefix is the value given to the --prefix option to configure). This prevents many rsh/ssh users from needing to modify their shell startup files to set the LD_LIBRARY_PATH for Open MPI (they will still need to set PATH or otherwise find the OMPI executables to mpicc/mpirun/etc. their MPI applications). Also added --noprefix option to orterun to disable this behavior. Finally, note that even if --enable-orterun-prefix-by-default is specified, if the user specifies --prefix or /path/to/mpirun, these options will override the default value of the prefix ($prefix). This commit was SVN r11669. The following Trac tickets were found above: Ticket 377 --> https://svn.open-mpi.org/trac/ompi/ticket/377
2006-09-15 06:52:08 +04:00
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Disable automatic --prefix behavior" },
Fix the app_context problem. NOTE: JEFF SHOULD CHECK THIS! I found that orterun was not tracking the index number of the app_contexts it was creating. Hence, the app_context->idx field was always sitting at zero. This index is used by the mapper to decide which app_context to use for each process - thus, with the value of each index being zero, the mapper only used the first app_context that was created. All others were ignored. Not sure when this might have gotten changed. Could be it was a problem that always existed, but didn't get exposed until something else was changed. Anyway, it seems to work now - could stand further testing. This commit was SVN r9389.
2006-03-23 19:53:11 +03:00
Add --show-progress option to mpirun This commit was SVN r21360.
2009-06-03 03:52:59 +04:00
{ "orte", "report", "launch_progress", '\0', "show-progress", "show-progress", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Output a brief periodic report on launch progress" },
Continue development of regular expression support by implementing it for slurm launches. Works for both initial (cmd line and non-cmd line) and comm_spawn launch. Additional work required to fully enable static port support when using cmd line regular expression launch system. This commit was SVN r21502.
2009-06-24 00:25:38 +04:00
{ "orte", "use", "regexp", '\0', "use-regexp", "use-regexp", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Use regular expressions for launch" },
Enable a new command line option to --report-events that instructs mpirun to RML-report specific events during job life to the requestor. This commit was SVN r21954.
2009-09-09 09:28:45 +04:00
{ "orte", "report", "events", '\0', "report-events", "report-events", 1,
NULL, OPAL_CMD_LINE_TYPE_STRING,
"Report events to a tool listening at the specified URI" },
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
{ "orte", "enable", "recovery", '\0', "enable-recovery", "enable-recovery", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Enable recovery from process failure [Default = disabled]" },
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
{ "orte", "max", "restarts", '\0', "max-restarts", "max-restarts", 1,
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
NULL, OPAL_CMD_LINE_TYPE_INT,
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
"Max number of times to restart a failed process" },
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#if OPAL_HAVE_HWLOC
Minimize the amount of topology info returned by the daemons. Most clusters, especially at scale, use the same node topology on every node, so there is no re ason to return the topology from every daemon. Borrow a page from the --hetero-apps page and let users indicate that the node topology differs by adding a -- hetero-nodes option to mpirun. If the option is set, then every daemon returns topology info. If not set, then only daemon vpid=1 returns it. We always want one daemon to return the topology as the head node is often different from the compute nodes. Having one daemon return the compute node topolo gy allows us to detect any such difference. All compute nodes are then set to the same topology. This commit was SVN r25408.
2011-11-01 22:43:10 +04:00
{ "orte", "hetero", "nodes", '\0', NULL, "hetero-nodes", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Nodes in cluster may differ in topology, so send the topology back from each node [Default = false]" },
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#endif
Minimize the amount of topology info returned by the daemons. Most clusters, especially at scale, use the same node topology on every node, so there is no re ason to return the topology from every daemon. Borrow a page from the --hetero-apps page and let users indicate that the node topology differs by adding a -- hetero-nodes option to mpirun. If the option is set, then every daemon returns topology info. If not set, then only daemon vpid=1 returns it. We always want one daemon to return the topology as the head node is often different from the compute nodes. Having one daemon return the compute node topolo gy allows us to detect any such difference. All compute nodes are then set to the same topology. This commit was SVN r25408.
2011-11-01 22:43:10 +04:00
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_CRDEBUG == 1
{ "opal", "cr", "enable_crdebug", '\0', "crdebug", "crdebug", 0,
NULL, OPAL_CMD_LINE_TYPE_BOOL,
"Enable C/R Debugging" },
#endif
Provide a convenience option to disable fault recovery (as opposed to setting three separate, long-named mca params) This commit was SVN r23327.
2010-07-01 23:31:11 +04:00
{ NULL, NULL, NULL, '\0', "disable-recovery", "disable-recovery", 0,
&orterun_globals.disable_recovery, OPAL_CMD_LINE_TYPE_BOOL,
"Disable recovery (resets all recovery options to off)" },
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* End of list */
{ NULL, NULL, NULL, '\0', NULL, NULL, 0,
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
NULL, OPAL_CMD_LINE_TYPE_NULL, NULL }
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
};
/*
* Local functions
*/
static int create_app(int argc, char* argv[], orte_app_context_t **app,
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
bool *made_app, char ***app_env);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
static int init_globals(void);
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
static int parse_globals(int argc, char* argv[], opal_cmd_line_t *cmd_line);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
static int parse_locals(int argc, char* argv[]);
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
static int parse_appfile(char *filename, char ***env);
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
static void run_debugger(char *basename, opal_cmd_line_t *cmd_line,
- Fixup several functions, that cannot return - Add one instance where we do not use a parameter in a function - Fix a buglet in commit r23689, where the attribute-for-function ptrs was applied. This commit was SVN r23690. The following SVN revision numbers were found above: r23689 --> open-mpi/ompi@5eb571c4586984bf8ec400c17898269349a0cca2
2010-08-31 16:21:13 +04:00
int argc, char *argv[], int num_procs) __opal_attribute_noreturn__;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
A little more clean-up. TotalView now works with --enable-debug build. Tested with: pls = rsh totalview.6.6.0-2 Linux cadillac82.ccstar.lanl.gov 2.4.24 #1 SMP Thu Jul 1 15:28:04 MDT 2004 i686 i686 i386 GNU/Linux This commit was SVN r7108.
2005-08-31 20:15:59 +04:00
int orterun(int argc, char *argv[])
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
{
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
int rc;
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
opal_cmd_line_t cmd_line;
Merge in tmp/jjh-scratch {{{ svn merge -r 18218:18240 https://svn.open-mpi.org/svn/ompi/tmp/jjh-scratch . }}} Contains: * Primarily a fix for a user reported problem where a cached file descriptor is causing a SIGPIPE on restart. * Cleanup some small memory leaks from using mca_base_param_env_var() - Thanks Jeff * Cleanup ORTE FT tool compilation in non-FT builds - Thanks Tim P. * Cleanup mpi interface with missplaced {{{OPAL_CR_ENTER_LIBRARY}}} - Thanks Terry * Some other sundry cleanup items all dealing with C/R functionality in the trunk. This commit was SVN r18241.
2008-04-23 04:17:12 +04:00
char * tmp_env_var = NULL;
Ensure all debugger interface symbols are present in orterun This commit was SVN r23823.
2010-10-01 01:36:00 +04:00
orte_debugger_breakpoint_fn_t foo;
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
orte_job_t *daemons;
Although we never really thought about it, we made an unconscious assumption in the mapper system - we assumed that the daemons would be placed on nodes in the order that the nodes appear in the allocation. In other words, we assumed that the launch environment would map processes in node order. Turns out, this isn't necessarily true. The Cray, for example, launches processes in a toroidal pattern, thus causing the daemons to wind up somewhere other than what we thought. Other environments (e.g., slurm) are also capable of such behavior, depending upon the default mapping algorithm they are told to use. Resolve this problem by making the daemon-to-node assignment in the affected environments when the daemon calls back and tells us what node it is on. Order the nodes in the mapping list so they are in daemon-vpid order as opposed to the order in which they show in the allocation. For environments that don't exhibit this mapping behavior (e.g., rsh), this won't have any impact. Also, clean up the vm launch procedure a little bit so it more closely aligns with the state machine implementation that is coming, and remove some lingering "slave" code. This commit was SVN r25551.
2011-11-30 23:58:24 +04:00
int32_t ljob;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
orte_app_context_t *app, *dapp;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
/* find our basename (the name of the executable) so that we can
use it in pretty-print error messages */
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename = opal_basename(argv[0]);
Merge the /tmp/jms-installdirs-trunk branch into the trunk. This finally brings in functionality that is already on the 1.2 branch, and was developed and tested in the v1.2ofed branch (and other places). Short version of new features: * Support for ibv_fork_init() * Automatically fill in the openib BTL bandwidth value by querying the HCA port * Installdirs functionality * Fixes to always use -I in the Fortran wrapper compilers (#924) * Gleb's mpool updates * Remove some kruft in btl/openib/configure.m4, therefore fixing the harmless warnings noted in #665 * Bunches of updates to the Linux RPM spec file I.e., effectively the same thing that r14411 brought to the v1.2 branch. Also effectively brought in r14432 and r14433 (some fixes on top of the original r14411 commit to v1.2). Still need to bring in the moral equivalent of r14445 after this commit (fixes to installdirs). This commit was SVN r14449. The following SVN revision numbers were found above: r14411 --> open-mpi/ompi@83b31314ae25efa4a03dfc560b569322a18e3d23 r14432 --> open-mpi/ompi@a48f160595abf35f5826e9370135a9056d4da486 r14433 --> open-mpi/ompi@68f346d2bc482ae29df0cdfe82171dcdb28e4a01 r14445 --> open-mpi/ompi@13d366b827719ec7078db17315513d5c6291ab96
2007-04-21 04:15:05 +04:00
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
/* Setup and parse the command line */
init_globals();
opal_cmd_line_create(&cmd_line, cmd_line_init);
mca_base_cmd_line_setup(&cmd_line);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
if (ORTE_SUCCESS != (rc = opal_cmd_line_parse(&cmd_line, true,
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
argc, argv)) ) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return rc;
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
}
Merge the /tmp/jms-installdirs-trunk branch into the trunk. This finally brings in functionality that is already on the 1.2 branch, and was developed and tested in the v1.2ofed branch (and other places). Short version of new features: * Support for ibv_fork_init() * Automatically fill in the openib BTL bandwidth value by querying the HCA port * Installdirs functionality * Fixes to always use -I in the Fortran wrapper compilers (#924) * Gleb's mpool updates * Remove some kruft in btl/openib/configure.m4, therefore fixing the harmless warnings noted in #665 * Bunches of updates to the Linux RPM spec file I.e., effectively the same thing that r14411 brought to the v1.2 branch. Also effectively brought in r14432 and r14433 (some fixes on top of the original r14411 commit to v1.2). Still need to bring in the moral equivalent of r14445 after this commit (fixes to installdirs). This commit was SVN r14449. The following SVN revision numbers were found above: r14411 --> open-mpi/ompi@83b31314ae25efa4a03dfc560b569322a18e3d23 r14432 --> open-mpi/ompi@a48f160595abf35f5826e9370135a9056d4da486 r14433 --> open-mpi/ompi@68f346d2bc482ae29df0cdfe82171dcdb28e4a01 r14445 --> open-mpi/ompi@13d366b827719ec7078db17315513d5c6291ab96
2007-04-21 04:15:05 +04:00
Enable modex-less launch. Consists of: 1. minor modification to include two new opal MCA params: (a) opal_profile: outputs what components were selected by each framework currently enabled for most, but not all, frameworks (b) opal_profile_file: name of file that contains profile info required for modex 2. introduction of two new tools: (a) ompi-probe: MPI process that simply calls MPI_Init/Finalize with opal_profile set. Also reports back the rml IP address for all interfaces on the node (b) ompi-profiler: uses ompi-probe to create the profile_file, also reports out a summary of what framework components are actually being used to help with configuration options 3. modification of the grpcomm basic component to utilize the profile file in place of the modex where possible 4. modification of orterun so it properly sees opal mca params and handles opal_profile correctly to ensure we don't get its profile 5. similar mod to orted as for orterun 6. addition of new test that calls orte_init followed by calls to grpcomm.barrier This is all completely benign unless actively selected. At the moment, it only supports modex-less launch for openib-based systems. Minor mod to the TCP btl would be required to enable it as well, if people are interested. Similarly, anyone interested in enabling other BTL's for modex-less operation should let me know and I'll give you the magic details. This seems to significantly improve scalability provided the file can be locally located on the nodes. I'm looking at an alternative means of disseminating the info (perhaps in launch message) as an option for removing that constraint. This commit was SVN r20098.
2008-12-10 02:49:02 +03:00
/*
* Since this process can now handle MCA/GMCA parameters, make sure to
* process them.
*/
mca_base_cmd_line_process_args(&cmd_line, &environ, &environ);
/* Ensure that enough of OPAL is setup for us to be able to run */
/*
* NOTE: (JJH)
* We need to allow 'mca_base_cmd_line_process_args()' to process command
* line arguments *before* calling opal_init_util() since the command
* line could contain MCA parameters that affect the way opal_init_util()
* functions. AMCA parameters are one such option normally received on the
* command line that affect the way opal_init_util() behaves.
* It is "safe" to call mca_base_cmd_line_process_args() before
* opal_init_util() since mca_base_cmd_line_process_args() does *not*
* depend upon opal_init_util() functionality.
*/
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
/* Need to initialize OPAL so that install_dirs are filled in */
A patch from UTK to allow orte_init(), opal_init(), and associated friends also receive &argc and &argv (George asked Jeff to Ralph to review before committing). The thought is that passing argv and argc to opal/orte_init be useful to other projects outside of OMPI that are using OPAL and/or ORTE (especially in conjunction with some other bootstrapping code where it is helpful to modify argv). It's such a small thing that it's easy to apply here to make others' lives a little easier. Ask George for more details; I'm just the messenger. :-) Judging by the copyrights on this patch, it's been around for a while. :-) This commit was SVN r22260.
2009-12-04 03:51:15 +03:00
if (OPAL_SUCCESS != opal_init_util(&argc, &argv)) {
Ensure to check that opal_init_util() didn't fail. This commit was SVN r18453.
2008-05-19 15:58:48 +04:00
exit(1);
}
Fix a problem observed by Brian where processes launched local to mpirun lost their environment except for MCA params. The problem stemmed from no longer launching a local orted on the same node as mpirun. The orted would save and reuse the base environment. Mpirun didn't do that, and the odls was using the orted's globally saved environment (which wasn't being set). This fix establishes a globally accessible base launch environment that both the orted and mpirun can utilize. Since we now use that, we don't need to pass it to the odls_launch_proc function, so remove that param from the API (and modify all components to handle the change). This commit was SVN r15405.
2007-07-13 19:47:57 +04:00
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
/* flag that I am the HNP - needs to be done prior to
* registering params
*/
orte_process_info.proc_type = ORTE_PROC_HNP;
Merge the /tmp/jms-installdirs-trunk branch into the trunk. This finally brings in functionality that is already on the 1.2 branch, and was developed and tested in the v1.2ofed branch (and other places). Short version of new features: * Support for ibv_fork_init() * Automatically fill in the openib BTL bandwidth value by querying the HCA port * Installdirs functionality * Fixes to always use -I in the Fortran wrapper compilers (#924) * Gleb's mpool updates * Remove some kruft in btl/openib/configure.m4, therefore fixing the harmless warnings noted in #665 * Bunches of updates to the Linux RPM spec file I.e., effectively the same thing that r14411 brought to the v1.2 branch. Also effectively brought in r14432 and r14433 (some fixes on top of the original r14411 commit to v1.2). Still need to bring in the moral equivalent of r14445 after this commit (fixes to installdirs). This commit was SVN r14449. The following SVN revision numbers were found above: r14411 --> open-mpi/ompi@83b31314ae25efa4a03dfc560b569322a18e3d23 r14432 --> open-mpi/ompi@a48f160595abf35f5826e9370135a9056d4da486 r14433 --> open-mpi/ompi@68f346d2bc482ae29df0cdfe82171dcdb28e4a01 r14445 --> open-mpi/ompi@13d366b827719ec7078db17315513d5c6291ab96
2007-04-21 04:15:05 +04:00
/* Setup MCA params */
Ensure params are registered prior to parsing global cmd line options in orterun so that debugger options are properly captured and acted upon. Ensure that routes to remote procs are set on the HNP before completing launch so that the debugger message can be sent. Solves a race condition that can exist in those environments where the HNP does not have local procs. This commit was SVN r18674.
2008-06-19 06:58:14 +04:00
orte_register_params();
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
Warn users if they set opal_paffinity_alone, either to true or false, that this parameter is no longer functional - they must use the --bind-to option and its corresponding mca param. This commit was SVN r25567.
2011-12-03 05:10:52 +04:00
/*** NOTIFY IF DEPRECATED OPAL_PAFFINITY_ALONE WAS SET ***/
if (opal_paffinity_alone) {
orte_show_help("help-opal-runtime.txt",
"opal_paffinity_alone:deprecated",
true);
}
Ensure the debugger symbols are loaded into orterun prior to orte_init so they are found by attaching debuggers. This commit was SVN r23368.
2010-07-09 21:51:00 +04:00
/* force the debugger symbols to be included in orterun.
* this is required since the symbols are instantiated in
* the orte library, yet they need to be accessed
* prior to orte_init when a debugger wants to launch
* us
*/
Fix debugger attach This commit was SVN r23921.
2010-10-23 00:07:24 +04:00
if (NULL == MPIR_proctable) {
rc = ORTE_SUCCESS;
}
if (0 == MPIR_proctable_size) {
rc = ORTE_SUCCESS;
}
Ensure all debugger interface symbols are present in orterun This commit was SVN r23823.
2010-10-01 01:36:00 +04:00
if (0 == MPIR_being_debugged) {
rc = ORTE_SUCCESS;
}
if (0 == MPIR_debug_state) {
rc = ORTE_SUCCESS;
}
if (0 == MPIR_i_am_starter) {
rc = ORTE_SUCCESS;
}
if (1 == MPIR_partial_attach_ok) {
rc = ORTE_SUCCESS;
}
Fix debugger attach This commit was SVN r23921.
2010-10-23 00:07:24 +04:00
if (NULL == MPIR_executable_path) {
rc = ORTE_SUCCESS;
}
if (NULL == MPIR_server_arguments) {
rc = ORTE_SUCCESS;
}
Ensure all debugger interface symbols are present in orterun This commit was SVN r23823.
2010-10-01 01:36:00 +04:00
if (0 == MPIR_forward_output) {
rc = ORTE_SUCCESS;
}
if (0 == MPIR_forward_comm) {
rc = ORTE_SUCCESS;
}
Add the appropriate hooks to tell Totalview to display the user's main program upon startup. Apparently, this hook got lost somewhere after the 1.2 series :-( Thanks to David Turner and the TV folks for passing this along. This commit was SVN r24549.
2011-03-21 20:40:58 +03:00
MPIR_force_to_main = 0;
Update some debugger attachment code per LLNL request This commit was SVN r23965.
2010-10-28 07:06:20 +04:00
memset(MPIR_attach_fifo, 0, MPIR_MAX_PATH_LENGTH);
Ensure all debugger interface symbols are present in orterun This commit was SVN r23823.
2010-10-01 01:36:00 +04:00
foo = MPIR_Breakpoint;
Ensure the debugger symbols are loaded into orterun prior to orte_init so they are found by attaching debuggers. This commit was SVN r23368.
2010-07-09 21:51:00 +04:00
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Check for some "global" command line params */
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
parse_globals(argc, argv, &cmd_line);
OBJ_DESTRUCT(&cmd_line);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* create a new job object to hold the info for this one - the
* jobid field will be filled in by the PLM when the job is
* launched
*/
jdata = OBJ_NEW(orte_job_t);
if (NULL == jdata) {
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* cannot call ORTE_ERROR_LOG as the errmgr
* hasn't been loaded yet!
*/
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return ORTE_ERR_OUT_OF_RESOURCE;
properly fix the usage of the app pointer array by checking for NULLs instead of forcing it to be the same size as the number of entries This commit was SVN r6395.
2005-07-08 22:48:25 +04:00
}
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
/* check what user wants us to do with stdin */
if (0 == strcmp(orterun_globals.stdin_target, "all")) {
jdata->stdin_target = ORTE_VPID_WILDCARD;
} else if (0 == strcmp(orterun_globals.stdin_target, "none")) {
jdata->stdin_target = ORTE_VPID_INVALID;
} else {
jdata->stdin_target = strtoul(orterun_globals.stdin_target, NULL, 10);
}
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* Parse each app, adding it to the job object */
parse_locals(argc, argv);
if (0 == jdata->num_apps) {
properly fix the usage of the app pointer array by checking for NULLs instead of forcing it to be the same size as the number of entries This commit was SVN r6395.
2005-07-08 22:48:25 +04:00
/* This should never happen -- this case should be caught in
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
create_app(), but let's just double check... */
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:nothing-to-do",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
true, orte_basename);
Define a default exit code for when things fail prior to a job launch - still needs work, but a start. Fix a deadlock loop when things really, really go bad. If we timeout trying to kill the job, then it's time to bail as cleanly as possible, not go back and keep trying. This commit was SVN r17715.
2008-03-05 04:46:30 +03:00
exit(ORTE_ERROR_DEFAULT_EXIT_CODE);
Updates to orterun: - Print error messages with the basename(argv[0]) rather than hard-coded argv[0] so that you can see an error message beginning with "mpirun" when you run mpirun, etc. - For all processes that died due to a signal: - If the signal was not SIGKILL, display the first N of them (where N defaults to 1) - If more than N processes died due to a non-SIGKILL signal, print "And X more processes aborted..." kind of message - Add --aborted command line parameter to change the default value of N - Also print out the total number of processes that died due to SIGKILL, with a disclaimer that it's impossible to know if we killed them or someone else killed them This commit was SVN r5406.
2005-04-16 01:52:58 +04:00
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Upgrade the ability of orterun to deal with cmd line MCA params that are passed to the orteds. Help reduce the size of the cmd line by eliminating duplicates where possible, and alert to duplicate entries that can cause problems. Add comments to both orterun and orted code explaining why we take a snapshot of the local environment and apply it to the local procs when they are spawned. This commit was SVN r18842.
2008-07-09 02:36:39 +04:00
/* save the environment for launch purposes. This MUST be
* done so that we can pass it to any local procs we
* spawn - otherwise, those local procs won't see any
* non-MCA envars were set in the enviro prior to calling
* orterun
*/
orte_launch_environ = opal_argv_copy(environ);
Ensure that any application-specific selection of ess module doesn't get overridden by what is given to the orted or orterun Cleanup tool name determination for CM This commit was SVN r22980.
2010-04-15 22:10:50 +04:00
/* purge an ess flag set externally */
opal_unsetenv("OMPI_MCA_ess", &orte_launch_environ);
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Merging in the jjhursey-ft-cr-stable branch (r13912 : HEAD). This merge adds Checkpoint/Restart support to Open MPI. The initial frameworks and components support a LAM/MPI-like implementation. This commit follows the risk assessment presented to the Open MPI core development group on Feb. 22, 2007. This commit closes trac:158 More details to follow. This commit was SVN r14051. The following SVN revisions from the original message are invalid or inconsistent and therefore were not cross-referenced: r13912 The following Trac tickets were found above: Ticket 158 --> https://svn.open-mpi.org/trac/ompi/ticket/158
2007-03-17 02:11:45 +03:00
/* Disable OPAL CR notifications for this tool */
opal_cr_set_enabled(false);
Merge in tmp/jjh-scratch {{{ svn merge -r 18218:18240 https://svn.open-mpi.org/svn/ompi/tmp/jjh-scratch . }}} Contains: * Primarily a fix for a user reported problem where a cached file descriptor is causing a SIGPIPE on restart. * Cleanup some small memory leaks from using mca_base_param_env_var() - Thanks Jeff * Cleanup ORTE FT tool compilation in non-FT builds - Thanks Tim P. * Cleanup mpi interface with missplaced {{{OPAL_CR_ENTER_LIBRARY}}} - Thanks Terry * Some other sundry cleanup items all dealing with C/R functionality in the trunk. This commit was SVN r18241.
2008-04-23 04:17:12 +04:00
tmp_env_var = mca_base_param_env_var("opal_cr_is_tool");
opal_setenv(tmp_env_var,
Move in some accumulated small features and minor bug fixes for C/R support. {{{ svn merge -r 16447:16475 https://svn.open-mpi.org/svn/ompi/tmp/jjh-fgs . }}} This commit was SVN r16478.
2007-10-17 17:47:36 +04:00
"1",
true, &environ);
Merge in tmp/jjh-scratch {{{ svn merge -r 18218:18240 https://svn.open-mpi.org/svn/ompi/tmp/jjh-scratch . }}} Contains: * Primarily a fix for a user reported problem where a cached file descriptor is causing a SIGPIPE on restart. * Cleanup some small memory leaks from using mca_base_param_env_var() - Thanks Jeff * Cleanup ORTE FT tool compilation in non-FT builds - Thanks Tim P. * Cleanup mpi interface with missplaced {{{OPAL_CR_ENTER_LIBRARY}}} - Thanks Terry * Some other sundry cleanup items all dealing with C/R functionality in the trunk. This commit was SVN r18241.
2008-04-23 04:17:12 +04:00
free(tmp_env_var);
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#endif
Revert this change, since even in the CR case we want to reset this var to NULL. Thanks to Jeff for the catch. This commit was SVN r22868.
2010-03-23 22:55:21 +03:00
tmp_env_var = NULL; /* Silence compiler warning */
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* Intialize our Open RTE environment
* Set the flag telling orte_init that I am NOT a
Okay, here is the massive checkin that restructures the registry trigger system for scalability. Actually, it isn't "quite" as large as it looks - it just touches a bunch of files. Also included is a fix to the attribute problem for singletons. Short explanation: The prior system placed triggers and subscriptions on the registry for each process - approximately eight/process. Each of these had to be checked every time there was a registry operation such as a "put" or "increment-value". For large numbers of processes, this repetitive checking consumed some significant time. The new system allows processes to "attach" to existing triggers and subscriptions, without creating a new one. Thus, there are now only eight triggers and five subscriptions on a job - *regardless of how many processes are being run*. This means that the registry now takes the same amount of time (which is pretty darn short) to process an operation regardless of how many processes are in a job. I'll provide some startup times from scalability tests shortly - need to complete the commit so I can move the system to an appropriate cluster. This commit was SVN r6164.
2005-06-24 20:59:37 +04:00
* singleton, but am "infrastructure" - prevents setting
* up incorrect infrastructure that only a singleton would
* require
*/
A patch from UTK to allow orte_init(), opal_init(), and associated friends also receive &argc and &argv (George asked Jeff to Ralph to review before committing). The thought is that passing argv and argc to opal/orte_init be useful to other projects outside of OMPI that are using OPAL and/or ORTE (especially in conjunction with some other bootstrapping code where it is helpful to modify argv). It's such a small thing that it's easy to apply here to make others' lives a little easier. Ask George for more details; I'm just the messenger. :-) Judging by the copyrights on this patch, it's been around for a while. :-) This commit was SVN r22260.
2009-12-04 03:51:15 +03:00
if (ORTE_SUCCESS != (rc = orte_init(&argc, &argv, ORTE_PROC_HNP))) {
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* cannot call ORTE_ERROR_LOG as it could be the errmgr
* never got loaded!
*/
This is actually a much smaller commit than it appears at first glance - it just touches a lot of files. The --without-rte-support configuration option has never really been implemented completely. The option caused various objects not to be defined and conditionally compiled some base functions, but did nothing to prevent build of the component libraries. Unfortunately, since many of those components use objects covered by the option, it caused builds to break if those components were allowed to build. Brian dealt with this in the past by creating platform files and using "no-build" to block the components. This was clunky, but acceptable when only one organization was using that option. However, that number has now expanded to at least two more locations. Accordingly, make --without-rte-support actually work by adding appropriate configury to prevent components from building when they shouldn't. While doing so, remove two frameworks (db and rmcast) that are no longer used as ORCM comes to a close (besides, they belonged in ORCM now anyway). Do some minor cleanups along the way. This commit was SVN r25497.
2011-11-23 01:24:35 +04:00
fprintf(stderr, "FAILED ORTE INIT\n");
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
return rc;
If we don't release the OPAL utils explicitly there will be a memory leak. This commit was SVN r24505.
2011-03-10 03:42:28 +03:00
}
After a nice, calm, rational discussion between Brian, Jeff, and myself, we decided to revert r24864 and r24862 to restore the reference counters in opal_init/finalize. The rationale was that we should instead change orte_init/finalize to also use reference counters to support multi-embedded libraries. Jeff and Brian will discuss proposing a similar change to mpi_init/finalize to the MPI Forum so that all three libraries will behave in similar manners. It was agreed that opal_init_util had wound up being used in unintended ways, which raised the problem of getting reference counts to work right. However, fixing it would involve more pain than it was worth - and so long as the other layers are made to behave similarly, I have no preference either way. Complete implementation will follow - for now, this just reverts the prior changes. This commit was SVN r24886. The following SVN revision numbers were found above: r24862 --> open-mpi/ompi@aa92e0c4eb16b2a6ec5b323befa325eaa1d1c10a r24864 --> open-mpi/ompi@a5062385c27647cfb75478daa9b79e5daee84fd5
2011-07-12 21:07:41 +04:00
/* finalize the OPAL utils. As they are opened again from orte_init->opal_init
* we continue to have a reference count on them. So we have to finalize them twice...
*/
opal_finalize_util();
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* get the daemon job object */
daemons = orte_get_job_data_object(ORTE_PROC_MY_NAME->jobid);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
/* check for request to report uri */
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
if (NULL != orterun_globals.report_uri) {
FILE *fp;
char *rml_uri;
rml_uri = orte_rml.get_contact_info();
if (0 == strcmp(orterun_globals.report_uri, "-")) {
/* if '-', then output to stdout */
printf("%s\n", (NULL == rml_uri) ? "NULL" : rml_uri);
} else if (0 == strcmp(orterun_globals.report_uri, "+")) {
/* if '+', output to stderr */
fprintf(stderr, "%s\n", (NULL == rml_uri) ? "NULL" : rml_uri);
} else {
fp = fopen(orterun_globals.report_uri, "w");
if (NULL == fp) {
orte_show_help("help-orterun.txt", "orterun:write_file", false,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, "uri", orterun_globals.report_uri);
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
exit(0);
}
fprintf(fp, "%s\n", (NULL == rml_uri) ? "NULL" : rml_uri);
fclose(fp);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
}
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
if (NULL != rml_uri) {
free(rml_uri);
}
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
}
Finish cleanup of stdin. Set non-stdio file descriptors to non-blocking (thanks to Jeff for catching that one). Handle writes that result in "would have blocked" errno. This commit was SVN r19793.
2008-10-24 05:42:58 +04:00
/* Change the default behavior of libevent such that we want to
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
continually block rather than blocking for the default timeout
and then looping around the progress engine again. There
should be nothing in the orted that cannot block in libevent
until "something" happens (i.e., there's no need to keep
cycling through progress because the only things that should
happen will happen in libevent). This is a minor optimization,
but what the heck... :-) */
Finish cleanup of stdin. Set non-stdio file descriptors to non-blocking (thanks to Jeff for catching that one). Handle writes that result in "would have blocked" errno. This commit was SVN r19793.
2008-10-24 05:42:58 +04:00
opal_progress_set_event_flag(OPAL_EVLOOP_ONCE);
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
/* If we have a prefix, then modify the PATH and
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
LD_LIBRARY_PATH environment variables in our copy. This
will ensure that any locally-spawned children will
have our executables and libraries in their path
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
For now, default to the prefix_dir provided in the first app_context.
Since there always MUST be at least one app_context, we are safe in
doing this.
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
*/
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
if (NULL != (app = (orte_app_context_t*)opal_pointer_array_get_item(jdata->apps, 0)) &&
NULL != app->prefix_dir) {
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
char *oldenv, *newenv, *lib_base, *bin_base;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* copy the prefix into the daemon job so that any launcher
* can find the orteds when we launch the virtual machine
*/
if (NULL == (dapp = (orte_app_context_t*)opal_pointer_array_get_item(daemons->apps, 0))) {
/* that's an error in the ess */
ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
return ORTE_ERR_NOT_FOUND;
}
dapp->prefix_dir = strdup(app->prefix_dir);
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
lib_base = opal_basename(opal_install_dirs.libdir);
bin_base = opal_basename(opal_install_dirs.bindir);
/* Reset PATH */
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
newenv = opal_os_path( false, app->prefix_dir, bin_base, NULL );
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
oldenv = getenv("PATH");
if (NULL != oldenv) {
char *temp;
asprintf(&temp, "%s:%s", newenv, oldenv );
free( newenv );
newenv = temp;
}
opal_setenv("PATH", newenv, true, &orte_launch_environ);
if (orte_debug_flag) {
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
opal_output(0, "%s: reset PATH: %s", orte_basename, newenv);
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
}
free(newenv);
free(bin_base);
/* Reset LD_LIBRARY_PATH */
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
newenv = opal_os_path( false, app->prefix_dir, lib_base, NULL );
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
oldenv = getenv("LD_LIBRARY_PATH");
if (NULL != oldenv) {
char* temp;
asprintf(&temp, "%s:%s", newenv, oldenv);
free(newenv);
newenv = temp;
}
opal_setenv("LD_LIBRARY_PATH", newenv, true, &orte_launch_environ);
if (orte_debug_flag) {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output(0, "%s: reset LD_LIBRARY_PATH: %s",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, newenv);
Ensure that the LD_LIBRARY_PATH and PATH get properly set for procs locally spawned by mpirun. This commit was SVN r15516.
2007-07-19 23:00:06 +04:00
}
free(newenv);
free(lib_base);
}
Push a generated "key" out to all the processes. This is necessary for some interconnect wireup in which all processes must agree on a "key" to initialize the interconnect with. This commit was SVN r11653.
2006-09-14 19:27:17 +04:00
/* pre-condition any network transports that require it */
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
if (ORTE_SUCCESS != (rc = orte_pre_condition_transports(jdata))) {
Push a generated "key" out to all the processes. This is necessary for some interconnect wireup in which all processes must agree on a "key" to initialize the interconnect with. This commit was SVN r11653.
2006-09-14 19:27:17 +04:00
ORTE_ERROR_LOG(rc);
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:precondition", false,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, NULL, NULL, rc);
Ensure that signals are de-trapped before exiting to stop the $#@@#$ event library from "asserting" This commit was SVN r20630.
2009-02-25 06:10:21 +03:00
ORTE_UPDATE_EXIT_STATUS(ORTE_ERROR_DEFAULT_EXIT_CODE);
goto DONE;
Push a generated "key" out to all the processes. This is necessary for some interconnect wireup in which all processes must agree on a "key" to initialize the interconnect with. This commit was SVN r11653.
2006-09-14 19:27:17 +04:00
}
Bring in an updated launch system for the orteds. This commit restores the ability to execute singletons and singleton comm_spawn, both in single node and multi-node environments. Short description: major changes include - 1. singletons now fork/exec a local daemon to manage their operations. 2. the orte daemon code now resides in libopen-rte 3. daemons no longer use the orte triggering system during startup. Instead, they directly call back to their parent pls component to report ready to operate. A base function to count the callbacks has been provided. I have modified all the pls components except xcpu and poe (don't understand either well enough to do it). Full functionality has been verified for rsh, SLURM, and TM systems. Compile has been verified for xgrid and gridengine. This commit was SVN r15390.
2007-07-12 23:53:18 +04:00
/* setup to listen for commands sent specifically to me, even though I would probably
* be the one sending them! Unfortunately, since I am a participating daemon,
* there are times I need to send a command to "all daemons", and that means *I* have
* to receive it too
*/
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
rc = orte_rml.recv_buffer_nb(ORTE_NAME_WILDCARD, ORTE_RML_TAG_DAEMON,
ORTE_RML_NON_PERSISTENT, orte_daemon_recv, NULL);
This is actually a much smaller commit than it appears at first glance - it just touches a lot of files. The --without-rte-support configuration option has never really been implemented completely. The option caused various objects not to be defined and conditionally compiled some base functions, but did nothing to prevent build of the component libraries. Unfortunately, since many of those components use objects covered by the option, it caused builds to break if those components were allowed to build. Brian dealt with this in the past by creating platform files and using "no-build" to block the components. This was clunky, but acceptable when only one organization was using that option. However, that number has now expanded to at least two more locations. Accordingly, make --without-rte-support actually work by adding appropriate configury to prevent components from building when they shouldn't. While doing so, remove two frameworks (db and rmcast) that are no longer used as ORCM comes to a close (besides, they belonged in ORCM now anyway). Do some minor cleanups along the way. This commit was SVN r25497.
2011-11-23 01:24:35 +04:00
if (rc != ORTE_SUCCESS && rc != ORTE_ERR_NOT_IMPLEMENTED) {
Bring in an updated launch system for the orteds. This commit restores the ability to execute singletons and singleton comm_spawn, both in single node and multi-node environments. Short description: major changes include - 1. singletons now fork/exec a local daemon to manage their operations. 2. the orte daemon code now resides in libopen-rte 3. daemons no longer use the orte triggering system during startup. Instead, they directly call back to their parent pls component to report ready to operate. A base function to count the callbacks has been provided. I have modified all the pls components except xcpu and poe (don't understand either well enough to do it). Full functionality has been verified for rsh, SLURM, and TM systems. Compile has been verified for xgrid and gridengine. This commit was SVN r15390.
2007-07-12 23:53:18 +04:00
ORTE_ERROR_LOG(rc);
Ensure that signals are de-trapped before exiting to stop the $#@@#$ event library from "asserting" This commit was SVN r20630.
2009-02-25 06:10:21 +03:00
ORTE_UPDATE_EXIT_STATUS(ORTE_ERROR_DEFAULT_EXIT_CODE);
goto DONE;
Bring in an updated launch system for the orteds. This commit restores the ability to execute singletons and singleton comm_spawn, both in single node and multi-node environments. Short description: major changes include - 1. singletons now fork/exec a local daemon to manage their operations. 2. the orte daemon code now resides in libopen-rte 3. daemons no longer use the orte triggering system during startup. Instead, they directly call back to their parent pls component to report ready to operate. A base function to count the callbacks has been provided. I have modified all the pls components except xcpu and poe (don't understand either well enough to do it). Full functionality has been verified for rsh, SLURM, and TM systems. Compile has been verified for xgrid and gridengine. This commit was SVN r15390.
2007-07-12 23:53:18 +04:00
}
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* setup the data server */
if (ORTE_SUCCESS != (rc = orte_data_server_init())) {
ORTE_ERROR_LOG(rc);
Ensure that signals are de-trapped before exiting to stop the $#@@#$ event library from "asserting" This commit was SVN r20630.
2009-02-25 06:10:21 +03:00
ORTE_UPDATE_EXIT_STATUS(ORTE_ERROR_DEFAULT_EXIT_CODE);
goto DONE;
Add a new option to orterun: "--do-not-launch" directs the system to do the allocation, map, job setup, etc., but don't actually launch the job. This lets us test all the setup portions of the code. Also, take the first step in updating how we handle mca params in ORTE - bring it closer to how it is done in the other two layers. Much more work to be done here. This commit was SVN r12838.
2006-12-13 07:51:38 +03:00
}
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Cleanup and fix bugs in the MPI dynamics section. Modify the dpm API so it properly takes ports instead of process names (as correctly identified by Aurelien). Fix race conditions in the use of ompi-server. Fix incompatibilities between the mpi bindings and the dpm implemenation that could cause segfaults due to uninitialized memory. Fix the ompi-server -h cmd line option so it actually tells you something! Add two new testing codes to the orte/test/mpi area: accept and connect. This commit was SVN r18176.
2008-04-16 18:27:42 +04:00
/* if an uri for the ompi-server was provided, set the route */
if (NULL != ompi_server) {
opal_buffer_t buf;
/* setup our route to the server */
OBJ_CONSTRUCT(&buf, opal_buffer_t);
opal_dss.pack(&buf, &ompi_server, 1, OPAL_STRING);
- Fix Coverity CID #15. This commit was SVN r21229.
2009-05-14 04:25:02 +04:00
if (ORTE_SUCCESS != (rc = orte_rml_base_update_contact_info(&buf))) {
ORTE_ERROR_LOG(rc);
ORTE_UPDATE_EXIT_STATUS(ORTE_ERROR_DEFAULT_EXIT_CODE);
goto DONE;
}
Cleanup and fix bugs in the MPI dynamics section. Modify the dpm API so it properly takes ports instead of process names (as correctly identified by Aurelien). Fix race conditions in the use of ompi-server. Fix incompatibilities between the mpi bindings and the dpm implemenation that could cause segfaults due to uninitialized memory. Fix the ompi-server -h cmd line option so it actually tells you something! Add two new testing codes to the orte/test/mpi area: accept and connect. This commit was SVN r18176.
2008-04-16 18:27:42 +04:00
OBJ_DESTRUCT(&buf);
Per the July technical meeting: During the discussion of MPI-2 functionality, it was pointed out by Aurelien that there was an inherent race condition between startup of ompi-server and mpirun. Specifically, if someone started ompi-server to run in the background as part of a script, and then immediately executed mpirun, it was possible that an MPI proc could attempt to contact the server (or that mpirun could try to read the server's contact file before the server is running and ready. At that time, we discussed createing a new tool "ompi-wait-server" that would wait for the server to be running, and/or probe to see if it is running and return true/false. However, rather than create yet another tool, it seemed just as effective to add the functionality to mpirun. Thus, this commit creates two new mpirun cmd line flags (hey, you can never have too many!): --wait-for-server : instructs mpirun to ping the server to see if it responds. This causes mpirun to execute an rml.ping to the server's URI with an appropriate timeout interval - if the ping isn't successful, mpirun attempts it again. --server-wait-time xx : sets the ping timeout interval to xx seconds. Note that mpirun will attempt to ping the server twice with this timeout, so we actually wait for twice this time. Default is 10 seconds, which should be plenty of time. This has only lightly been tested. It works if the server is present, and outputs a nice error message if it cannot be contacted. I have not tested the race condition case. This commit was SVN r19152.
2008-08-05 00:29:50 +04:00
/* check if we are to wait for the server to start - resolves
* a race condition that can occur when the server is run
* as a background job - e.g., in scripts
*/
if (orterun_globals.wait_for_server) {
/* ping the server */
struct timeval timeout;
timeout.tv_sec = orterun_globals.server_wait_timeout;
timeout.tv_usec = 0;
if (ORTE_SUCCESS != (rc = orte_rml.ping(ompi_server, &timeout))) {
/* try it one more time */
if (ORTE_SUCCESS != (rc = orte_rml.ping(ompi_server, &timeout))) {
/* okay give up */
orte_show_help("help-orterun.txt", "orterun:server-not-found", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, ompi_server,
Per the July technical meeting: During the discussion of MPI-2 functionality, it was pointed out by Aurelien that there was an inherent race condition between startup of ompi-server and mpirun. Specifically, if someone started ompi-server to run in the background as part of a script, and then immediately executed mpirun, it was possible that an MPI proc could attempt to contact the server (or that mpirun could try to read the server's contact file before the server is running and ready. At that time, we discussed createing a new tool "ompi-wait-server" that would wait for the server to be running, and/or probe to see if it is running and return true/false. However, rather than create yet another tool, it seemed just as effective to add the functionality to mpirun. Thus, this commit creates two new mpirun cmd line flags (hey, you can never have too many!): --wait-for-server : instructs mpirun to ping the server to see if it responds. This causes mpirun to execute an rml.ping to the server's URI with an appropriate timeout interval - if the ping isn't successful, mpirun attempts it again. --server-wait-time xx : sets the ping timeout interval to xx seconds. Note that mpirun will attempt to ping the server twice with this timeout, so we actually wait for twice this time. Default is 10 seconds, which should be plenty of time. This has only lightly been tested. It works if the server is present, and outputs a nice error message if it cannot be contacted. I have not tested the race condition case. This commit was SVN r19152.
2008-08-05 00:29:50 +04:00
(long)orterun_globals.server_wait_timeout,
ORTE_ERROR_NAME(rc));
Ensure that signals are de-trapped before exiting to stop the $#@@#$ event library from "asserting" This commit was SVN r20630.
2009-02-25 06:10:21 +03:00
ORTE_UPDATE_EXIT_STATUS(ORTE_ERROR_DEFAULT_EXIT_CODE);
Per the July technical meeting: During the discussion of MPI-2 functionality, it was pointed out by Aurelien that there was an inherent race condition between startup of ompi-server and mpirun. Specifically, if someone started ompi-server to run in the background as part of a script, and then immediately executed mpirun, it was possible that an MPI proc could attempt to contact the server (or that mpirun could try to read the server's contact file before the server is running and ready. At that time, we discussed createing a new tool "ompi-wait-server" that would wait for the server to be running, and/or probe to see if it is running and return true/false. However, rather than create yet another tool, it seemed just as effective to add the functionality to mpirun. Thus, this commit creates two new mpirun cmd line flags (hey, you can never have too many!): --wait-for-server : instructs mpirun to ping the server to see if it responds. This causes mpirun to execute an rml.ping to the server's URI with an appropriate timeout interval - if the ping isn't successful, mpirun attempts it again. --server-wait-time xx : sets the ping timeout interval to xx seconds. Note that mpirun will attempt to ping the server twice with this timeout, so we actually wait for twice this time. Default is 10 seconds, which should be plenty of time. This has only lightly been tested. It works if the server is present, and outputs a nice error message if it cannot be contacted. I have not tested the race condition case. This commit was SVN r19152.
2008-08-05 00:29:50 +04:00
goto DONE;
}
}
}
Cleanup and fix bugs in the MPI dynamics section. Modify the dpm API so it properly takes ports instead of process names (as correctly identified by Aurelien). Fix race conditions in the use of ompi-server. Fix incompatibilities between the mpi bindings and the dpm implemenation that could cause segfaults due to uninitialized memory. Fix the ompi-server -h cmd line option so it actually tells you something! Add two new testing codes to the orte/test/mpi area: accept and connect. This commit was SVN r18176.
2008-04-16 18:27:42 +04:00
}
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* we may need to look at the apps for the user's job
* to get our full list of nodes, so prep the job for
* launch - start by getting a jobid for it */
if (ORTE_SUCCESS != (rc = orte_plm_base_create_jobid(jdata))) {
ORTE_ERROR_LOG(rc);
goto DONE;
}
/* store it on the global job data pool - this is the key
* step required before we launch the daemons. It allows
* the orte_rmaps_base_setup_virtual_machine routine to
* search all apps for any hosts to be used by the vm
*/
ljob = ORTE_LOCAL_JOBID(jdata->jobid);
opal_pointer_array_set_item(orte_job_data, ljob, jdata);
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
Although we never really thought about it, we made an unconscious assumption in the mapper system - we assumed that the daemons would be placed on nodes in the order that the nodes appear in the allocation. In other words, we assumed that the launch environment would map processes in node order. Turns out, this isn't necessarily true. The Cray, for example, launches processes in a toroidal pattern, thus causing the daemons to wind up somewhere other than what we thought. Other environments (e.g., slurm) are also capable of such behavior, depending upon the default mapping algorithm they are told to use. Resolve this problem by making the daemon-to-node assignment in the affected environments when the daemon calls back and tells us what node it is on. Order the nodes in the mapping list so they are in daemon-vpid order as opposed to the order in which they show in the allocation. For environments that don't exhibit this mapping behavior (e.g., rsh), this won't have any impact. Also, clean up the vm launch procedure a little bit so it more closely aligns with the state machine implementation that is coming, and remove some lingering "slave" code. This commit was SVN r25551.
2011-11-30 23:58:24 +04:00
/* run the allocator on the application job - this allows us to
* pickup any host or hostfile arguments so we get the full
* array of nodes in our allocation
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
*/
Although we never really thought about it, we made an unconscious assumption in the mapper system - we assumed that the daemons would be placed on nodes in the order that the nodes appear in the allocation. In other words, we assumed that the launch environment would map processes in node order. Turns out, this isn't necessarily true. The Cray, for example, launches processes in a toroidal pattern, thus causing the daemons to wind up somewhere other than what we thought. Other environments (e.g., slurm) are also capable of such behavior, depending upon the default mapping algorithm they are told to use. Resolve this problem by making the daemon-to-node assignment in the affected environments when the daemon calls back and tells us what node it is on. Order the nodes in the mapping list so they are in daemon-vpid order as opposed to the order in which they show in the allocation. For environments that don't exhibit this mapping behavior (e.g., rsh), this won't have any impact. Also, clean up the vm launch procedure a little bit so it more closely aligns with the state machine implementation that is coming, and remove some lingering "slave" code. This commit was SVN r25551.
2011-11-30 23:58:24 +04:00
if (ORTE_SUCCESS != (rc = orte_ras.allocate(jdata))) {
goto DONE;
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
}
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* Spawn the job */
rc = orte_plm.spawn(jdata);
Fix the app-failed-to-start capability that was broken by r14554 (holding the caller in rmgr.spawn until the application - as opposed to just the orteds - have started). Allow the rmgr.spawn function to return if the app terminates, correctly handling its return status code to show abnormal termination. Modify orterun to correctly handle the returned status code so it doesn't enter a conditioned wait if the app fails to start since it will never wakeup if it does. This commit was SVN r14693. The following SVN revision numbers were found above: r14554 --> open-mpi/ompi@4510b426388a3f8c311fdd7bea1a1c9187bcf77b
2007-05-18 17:29:11 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* now wait until the termination event fires */
Convert the opal_event framework to use direct function calls instead of hiding functions behind function pointers. Eliminate the opal_object_t abstraction of libevent's event struct so it can be directly passed to the libevent functions. Note: the ompi_check_libfca.m4 file had to be modified to avoid it stomping on global CPPFLAGS and the like. The file was also relocated to the ompi/config directory as it pertains solely to an ompi-layer component. Forgive the mid-day configure change, but I know Shiqing is working the windows issues and don't want to cause him unnecessary redo work. This commit was SVN r23966.
2010-10-28 19:22:46 +04:00
opal_event_dispatch(opal_event_base);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* we only reach this point by jumping there due
* to an error - so just cleanup and leave
*/
Enable VM launch. For some time, ORTE has had the ability to launch daemons on all nodes prior to launching an application. It has largely been used outside of the OMPI community, and so was never explicitly turned "on" inside OMPI releases. Nevertheless, the code has been there. Allowing VM launches does not require ANY changes to existing PLM components. All that was required was to have orterun launch the daemons as a separate call to orte_plm.spawn -prior- to launching the applications. The rest of the VM support code resides in the rmaps framework: (a) a check when asked to map a job to see if it is the daemon job, and (b) a separate "setup_virtual_machine" mapper in the rmaps base that creates the required map so the PLM's will do the right thing. In order to support those users who have no RM allocation but like to give the allocation in the form of a -host or -hostfile argument to their application, there is a little more code in orterun and the setup_virtual_machine mapper to capture information passed in that manner. This has been tested with rsh and slurm environments, and, since there is nothing environment-specific in the implementation, should work in others as well - but needs to be proven. This commit was SVN r24524.
2011-03-13 01:50:53 +03:00
DONE:
Now that we understand why it failed before (thanks George!), go back to relaying cmds before processing them on the orteds so we pickup some speed by executing cmds in parallel. Modify the termination system accordingly. Cleanup the termination in orterun when abnormally commanded via ctrl-c. We can just directly terminate_orteds as the orteds always kill any lingering local procs before exiting - no need to do the two-step cha-cha. This commit was SVN r21123.
2009-04-30 19:08:02 +04:00
ORTE_UPDATE_EXIT_STATUS(orte_exit_status);
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_quit();
If we don't release the OPAL utils explicitly there will be a memory leak. This commit was SVN r24505.
2011-03-10 03:42:28 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return orte_exit_status;
}
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
static int init_globals(void)
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
{
* More changes from the tim branch. Still has problems with ABORTed procs, but now tells you when it can't find orted. Also includes memory leak plugs, bproc fixes, and gm repairs. This commit was SVN r4937.
2005-03-19 02:58:36 +03:00
/* Only CONSTRUCT things once */
if (!globals_init) {
* rename locking code from ompi to opal This commit was SVN r6327.
2005-07-04 02:45:48 +04:00
OBJ_CONSTRUCT(&orterun_globals.lock, opal_mutex_t);
Do not completely reset the orterun_globals. Keep the condition and the mutex, but reset everything else. Once initialized the condition (and the attached mutex) should be kept alive as long as possible if we want to be able to retrieve all the informations. This commit was SVN r12253.
2006-10-23 07:34:08 +04:00
orterun_globals.env_val = NULL;
orterun_globals.appfile = NULL;
orterun_globals.wdir = NULL;
orterun_globals.path = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
orterun_globals.ompi_server = NULL;
Per the July technical meeting: During the discussion of MPI-2 functionality, it was pointed out by Aurelien that there was an inherent race condition between startup of ompi-server and mpirun. Specifically, if someone started ompi-server to run in the background as part of a script, and then immediately executed mpirun, it was possible that an MPI proc could attempt to contact the server (or that mpirun could try to read the server's contact file before the server is running and ready. At that time, we discussed createing a new tool "ompi-wait-server" that would wait for the server to be running, and/or probe to see if it is running and return true/false. However, rather than create yet another tool, it seemed just as effective to add the functionality to mpirun. Thus, this commit creates two new mpirun cmd line flags (hey, you can never have too many!): --wait-for-server : instructs mpirun to ping the server to see if it responds. This causes mpirun to execute an rml.ping to the server's URI with an appropriate timeout interval - if the ping isn't successful, mpirun attempts it again. --server-wait-time xx : sets the ping timeout interval to xx seconds. Note that mpirun will attempt to ping the server twice with this timeout, so we actually wait for twice this time. Default is 10 seconds, which should be plenty of time. This has only lightly been tested. It works if the server is present, and outputs a nice error message if it cannot be contacted. I have not tested the race condition case. This commit was SVN r19152.
2008-08-05 00:29:50 +04:00
orterun_globals.wait_for_server = false;
orterun_globals.server_wait_timeout = 10;
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
orterun_globals.stdin_target = "0";
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
orterun_globals.report_pid = NULL;
orterun_globals.report_uri = NULL;
Provide a convenience option to disable fault recovery (as opposed to setting three separate, long-named mca params) This commit was SVN r23327.
2010-07-01 23:31:11 +04:00
orterun_globals.disable_recovery = false;
* More changes from the tim branch. Still has problems with ABORTed procs, but now tells you when it can't find orted. Also includes memory leak plugs, bproc fixes, and gm repairs. This commit was SVN r4937.
2005-03-19 02:58:36 +03:00
}
Per LANL's stated need, add functionality that runs a.out across ALL available process slots if no num_proc is specified on the command line. However, please note the following limitation: we ONLY allow ONE application to be specified on the command line when this feature is invoked. If multiple apps are specified, the user MUST also specify the number to be launched for each and every one of them. Update the help text to report errors when not following that rule. Also updated the RMAPS help text to reflect the reorganization of some of the round-robin code into the base. The new functionality has been tested under Mac OS-X and on Odin using an MPI program. Both byslot and bynode mapping have been checked and verified. Operational support for other systems needs to be verified - I respectfully request people's help in doing so. This commit was SVN r10708.
2006-07-11 01:25:33 +04:00
/* Reset the other fields every time */
* More changes from the tim branch. Still has problems with ABORTed procs, but now tells you when it can't find orted. Also includes memory leak plugs, bproc fixes, and gm repairs. This commit was SVN r4937.
2005-03-19 02:58:36 +03:00
Do not completely reset the orterun_globals. Keep the condition and the mutex, but reset everything else. Once initialized the condition (and the attached mutex) should be kept alive as long as possible if we want to be able to retrieve all the informations. This commit was SVN r12253.
2006-10-23 07:34:08 +04:00
orterun_globals.help = false;
orterun_globals.version = false;
orterun_globals.verbose = false;
orterun_globals.debugger = false;
Extend the mpirun options to support "--npernode N". This option tells the system to spawn N procs/node across all nodes in the allocation. If N is greater than the number of allocated slots, then the usual oversubscription logic will apply (i.e., the system will error out if oversubscription is not allowed, otherwise it will run with the sched_yield set to non-aggressive behavior). In "--npernode" operation, the "-np" command line parameter is ignored. This commit was SVN r12826.
2006-12-12 03:54:05 +03:00
orterun_globals.num_procs = 0;
My eyes may be deceiving me....but I do believe these comparisons are backwards! I think we only really want to "free" these variables if they are NOT NULL - as opposed to "free"ing them if they ARE NULL. This commit was SVN r12612.
2006-11-16 01:59:01 +03:00
if( NULL != orterun_globals.env_val )
Do not completely reset the orterun_globals. Keep the condition and the mutex, but reset everything else. Once initialized the condition (and the attached mutex) should be kept alive as long as possible if we want to be able to retrieve all the informations. This commit was SVN r12253.
2006-10-23 07:34:08 +04:00
free( orterun_globals.env_val );
orterun_globals.env_val = NULL;
My eyes may be deceiving me....but I do believe these comparisons are backwards! I think we only really want to "free" these variables if they are NOT NULL - as opposed to "free"ing them if they ARE NULL. This commit was SVN r12612.
2006-11-16 01:59:01 +03:00
if( NULL != orterun_globals.appfile )
Do not completely reset the orterun_globals. Keep the condition and the mutex, but reset everything else. Once initialized the condition (and the attached mutex) should be kept alive as long as possible if we want to be able to retrieve all the informations. This commit was SVN r12253.
2006-10-23 07:34:08 +04:00
free( orterun_globals.appfile );
orterun_globals.appfile = NULL;
My eyes may be deceiving me....but I do believe these comparisons are backwards! I think we only really want to "free" these variables if they are NOT NULL - as opposed to "free"ing them if they ARE NULL. This commit was SVN r12612.
2006-11-16 01:59:01 +03:00
if( NULL != orterun_globals.wdir )
Do not completely reset the orterun_globals. Keep the condition and the mutex, but reset everything else. Once initialized the condition (and the attached mutex) should be kept alive as long as possible if we want to be able to retrieve all the informations. This commit was SVN r12253.
2006-10-23 07:34:08 +04:00
free( orterun_globals.wdir );
orterun_globals.wdir = NULL;
if( NULL != orterun_globals.path )
free( orterun_globals.path );
orterun_globals.path = NULL;
* More changes from the tim branch. Still has problems with ABORTed procs, but now tells you when it can't find orted. Also includes memory leak plugs, bproc fixes, and gm repairs. This commit was SVN r4937.
2005-03-19 02:58:36 +03:00
Merging in the jjhursey-ft-cr-stable branch (r13912 : HEAD). This merge adds Checkpoint/Restart support to Open MPI. The initial frameworks and components support a LAM/MPI-like implementation. This commit follows the risk assessment presented to the Open MPI core development group on Feb. 22, 2007. This commit closes trac:158 More details to follow. This commit was SVN r14051. The following SVN revisions from the original message are invalid or inconsistent and therefore were not cross-referenced: r13912 The following Trac tickets were found above: Ticket 158 --> https://svn.open-mpi.org/trac/ompi/ticket/158
2007-03-17 02:11:45 +03:00
orterun_globals.preload_binary = false;
orterun_globals.preload_files = NULL;
orterun_globals.preload_files_dest_dir = NULL;
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_FT_CR == 1
orterun_globals.sstore_load = NULL;
#endif
* More changes from the tim branch. Still has problems with ABORTed procs, but now tells you when it can't find orted. Also includes memory leak plugs, bproc fixes, and gm repairs. This commit was SVN r4937.
2005-03-19 02:58:36 +03:00
/* All done */
globals_init = true;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
return ORTE_SUCCESS;
}
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
static int parse_globals(int argc, char* argv[], opal_cmd_line_t *cmd_line)
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
{
Fix for ticket #91 mpirun/orterun now has an option to print the version number. If -V/--version is given, it will print the version number. If it's the only option, we exit cleanly. Otherwise, we continue on as if --version wasn't given (except we've printed the version number). --This line, and th se below, will be ignored-- M orte/tools/orterun/orterun.c M orte/tools/orterun/help-orterun.txt This commit was SVN r10276.
2006-06-09 21:21:23 +04:00
/* print version if requested. Do this before check for help so
that --version --help works as one might expect. */
* Per discussion at quarterly meeting and bug #91, print out the bug contact point when printing version and help strings This commit was SVN r10484.
2006-06-22 23:48:27 +04:00
if (orterun_globals.version &&
!(1 == argc || orterun_globals.help)) {
Fix for ticket #91 mpirun/orterun now has an option to print the version number. If -V/--version is given, it will print the version number. If it's the only option, we exit cleanly. Otherwise, we continue on as if --version wasn't given (except we've printed the version number). --This line, and th se below, will be ignored-- M orte/tools/orterun/orterun.c M orte/tools/orterun/help-orterun.txt This commit was SVN r10276.
2006-06-09 21:21:23 +04:00
char *project_name = NULL;
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
if (0 == strcmp(orte_basename, "mpirun")) {
Fix for ticket #91 mpirun/orterun now has an option to print the version number. If -V/--version is given, it will print the version number. If it's the only option, we exit cleanly. Otherwise, we continue on as if --version wasn't given (except we've printed the version number). --This line, and th se below, will be ignored-- M orte/tools/orterun/orterun.c M orte/tools/orterun/help-orterun.txt This commit was SVN r10276.
2006-06-09 21:21:23 +04:00
project_name = "Open MPI";
} else {
project_name = "OpenRTE";
}
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:version", false,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, project_name, OPAL_VERSION,
* Per discussion at quarterly meeting and bug #91, print out the bug contact point when printing version and help strings This commit was SVN r10484.
2006-06-22 23:48:27 +04:00
PACKAGE_BUGREPORT);
Fix for ticket #91 mpirun/orterun now has an option to print the version number. If -V/--version is given, it will print the version number. If it's the only option, we exit cleanly. Otherwise, we continue on as if --version wasn't given (except we've printed the version number). --This line, and th se below, will be ignored-- M orte/tools/orterun/orterun.c M orte/tools/orterun/help-orterun.txt This commit was SVN r10276.
2006-06-09 21:21:23 +04:00
/* if we were the only argument, exit */
if (2 == argc) exit(0);
}
removed the version command line option. Added some more user help messages This commit was SVN r6672.
2005-07-29 01:17:48 +04:00
/* Check for help request */
Fix two annoying things about orterun: - only show the basename of the argv[0] when displaying error messages - if orterun is run with no arguments, show the help message This commit was SVN r5269.
2005-04-12 20:01:30 +04:00
if (1 == argc || orterun_globals.help) {
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
char *args = NULL;
* Per discussion at quarterly meeting and bug #91, print out the bug contact point when printing version and help strings This commit was SVN r10484.
2006-06-22 23:48:27 +04:00
char *project_name = NULL;
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
if (0 == strcmp(orte_basename, "mpirun")) {
* Per discussion at quarterly meeting and bug #91, print out the bug contact point when printing version and help strings This commit was SVN r10484.
2006-06-22 23:48:27 +04:00
project_name = "Open MPI";
} else {
project_name = "OpenRTE";
}
This commit cleans up the AMCA parameter implementation a bit. * Remove the 'opal_mca_base_param_use_amca_sets' global variable * Harness the fact that you can (read should) call the cmd_line functions before initializing opal_init_util(). This pushes the MCA/GMCA/AMCA command line options into the environment before OPAL inits and starts to use these values. By putting the cmd_line parse before opal_init_util in orterun and orted we only parse the *MCA parameter files once, and correctly (alleviating the need to 'recache' the files on init.) * Small bits of cleanup. This commit was SVN r15219.
2007-06-27 05:03:31 +04:00
args = opal_cmd_line_get_usage_msg(cmd_line);
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:usage", false,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, project_name, OPAL_VERSION,
orte_basename, args,
* Per discussion at quarterly meeting and bug #91, print out the bug contact point when printing version and help strings This commit was SVN r10484.
2006-06-22 23:48:27 +04:00
PACKAGE_BUGREPORT);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
free(args);
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* If someone asks for help, that should be all we do */
exit(0);
}
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
/* check for request to report pid */
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
if (NULL != orterun_globals.report_pid) {
FILE *fp;
if (0 == strcmp(orterun_globals.report_pid, "-")) {
/* if '-', then output to stdout */
printf("%d\n", (int)getpid());
} else if (0 == strcmp(orterun_globals.report_pid, "+")) {
/* if '+', output to stderr */
fprintf(stderr, "%d\n", (int)getpid());
} else {
fp = fopen(orterun_globals.report_pid, "w");
if (NULL == fp) {
orte_show_help("help-orterun.txt", "orterun:write_file", false,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, "pid", orterun_globals.report_pid);
Per request from Aurelien, make orterun report-pid and report-uri functions work the same as that of ompi-server. Since these are used for ompi-server-like functionality, it makes sense that the report options work the same. Make orte-top take the corresponding input the same way too for consistency. The modified cmd line options are: --report-uri x where x is either '-' for stdout, '+' for stderr, or a filename --report-pid x where x is the same as above For orte-top, you can now provide either a pid or a uri (which allows connection to remote mpiruns), specified either directly or with a "file:x" option as per mpirun's ompi-server option. Note: I did not add a report-pid option to ompi-server as it probably wouldn't be useful - the report-uri option works as well, and allows remote access (which is likely the normal way it would be used). This commit was SVN r20168.
2008-12-24 18:27:46 +03:00
exit(0);
}
fprintf(fp, "%d\n", (int)getpid());
fclose(fp);
}
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
}
Good weekend brainless activity -- implement the orterun command line debugger scheme described in http://www.open-mpi.org/community/lists/users/2005/10/0214.php. This makes our user-level debugger scheme much more vendor-independent (although the "-tv" option will still work for backwards compatibility -- it'll just be a synonum of "--debug"). This commit was SVN r8206.
2005-11-20 19:06:53 +03:00
/* Do we want a user-level debugger? */
Add "-tv" option to orterun: orterun -tv -np 4 foo which will turn around and re-exec: totalview orterun -a -np 4 foo This commit was SVN r7636.
2005-10-05 14:24:34 +04:00
Good weekend brainless activity -- implement the orterun command line debugger scheme described in http://www.open-mpi.org/community/lists/users/2005/10/0214.php. This makes our user-level debugger scheme much more vendor-independent (although the "-tv" option will still work for backwards compatibility -- it'll just be a synonum of "--debug"). This commit was SVN r8206.
2005-11-20 19:06:53 +03:00
if (orterun_globals.debugger) {
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
run_debugger(orte_basename, cmd_line, argc, argv, orterun_globals.num_procs);
Good weekend brainless activity -- implement the orterun command line debugger scheme described in http://www.open-mpi.org/community/lists/users/2005/10/0214.php. This makes our user-level debugger scheme much more vendor-independent (although the "-tv" option will still work for backwards compatibility -- it'll just be a synonum of "--debug"). This commit was SVN r8206.
2005-11-20 19:06:53 +03:00
}
Add "-tv" option to orterun: orterun -tv -np 4 foo which will turn around and re-exec: totalview orterun -a -np 4 foo This commit was SVN r7636.
2005-10-05 14:24:34 +04:00
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* if recovery was disabled on the cmd line, do so */
Provide a convenience option to disable fault recovery (as opposed to setting three separate, long-named mca params) This commit was SVN r23327.
2010-07-01 23:31:11 +04:00
if (orterun_globals.disable_recovery) {
orte_enable_recovery = false;
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
orte_max_restarts = 0;
Provide a convenience option to disable fault recovery (as opposed to setting three separate, long-named mca params) This commit was SVN r23327.
2010-07-01 23:31:11 +04:00
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
return ORTE_SUCCESS;
}
static int parse_locals(int argc, char* argv[])
{
int i, rc, app_num;
int temp_argc;
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
char **temp_argv, **env;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
orte_app_context_t *app;
bool made_app;
With the branch to 1.2 made.... Clean up the remainder of the size_t references in the runtime itself. Convert to orte_std_cntr_t wherever it makes sense (only avoid those places where the actual memory size is referenced). Remove the obsolete oob barrier function (we actually obsoleted it a long time ago - just never bothered to clean it up). I have done my best to go through all the components and catch everything, even if I couldn't test compile them since I wasn't on that type of system. Still, I cannot guarantee that problems won't show up when you test this on specific systems. Usually, these will just show as "warning: comparison between signed and unsigned" notes which are easily fixed (just change a size_t to orte_std_cntr_t). In some places, people didn't use size_t, but instead used some other variant (e.g., I found several places with uint32_t). I tried to catch all of them, but... Once we get all the instances caught and fixed, this should once and for all resolve many of the heterogeneity problems. This commit was SVN r11204.
2006-08-15 23:54:10 +04:00
orte_std_cntr_t j, size1;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* if the ompi-server was given, then set it up here */
if (NULL != orterun_globals.ompi_server) {
/* someone could have passed us a file instead of a uri, so
* we need to first check to see what we have - if it starts
* with "file", then we know it is a file. Otherwise, we assume
* it is a uri as provided by the ompi-server's output
* of an ORTE-standard string. Note that this is NOT a standard
* uri as it starts with the process name!
*/
Repair the MPI-2 dynamic operations. This includes: 1. repair of the linear and direct routed modules 2. repair of the ompi/pubsub/orte module to correctly init routes to the ompi-server, and correctly handle failure to correctly parse the provided ompi-server URI 3. modification of orterun to accept both "file" and "FILE" for designating where the ompi-server URI is to be found - purely a convenience feature 4. resolution of a message ordering problem during the connect/accept handshake that allowed the "send-first" proc to attempt to send to the "recv-first" proc before the HNP had actually updated its routes. Let this be a further reminder to all - message ordering is NOT guaranteed in the OOB 5. Repair the ompi/dpm/orte module to correctly init routes during connect/accept. Reminder to all: messages sent to procs in another job family (i.e., started by a different mpirun) are ALWAYS routed through the respective HNPs. As per the comments in orte/routed, this is REQUIRED to maintain connect/accept (where only the root proc on each side is capable of init'ing the routes), allow communication between mpirun's using different routing modules, and to minimize connections on tools such as ompi-server. It is all taken care of "under the covers" by the OOB to ensure that a route back to the sender is maintained, even when the different mpirun's are using different routed modules. 6. corrections in the orte/odls to ensure proper identification of daemons participating in a dynamic launch 7. corrections in build/nidmap to support update of an existing nidmap during dynamic launch 8. corrected implementation of the update_arch function in the ESS, along with consolidation of a number of ESS operations into base functions for easier maintenance. The ability to support info from multiple jobs was added, although we don't currently do so - this will come later to support further fault recovery strategies 9. minor updates to several functions to remove unnecessary and/or no longer used variables and envar's, add some debugging output, etc. 10. addition of a new macro ORTE_PROC_IS_DAEMON that resolves to true if the provided proc is a daemon There is still more cleanup to be done for efficiency, but this at least works. Tested on single-node Mac, multi-node SLURM via odin. Tests included connect/accept, publish/lookup/unpublish, comm_spawn, comm_spawn_multiple, and singleton comm_spawn. Fixes ticket #1256 This commit was SVN r18804.
2008-07-03 21:53:37 +04:00
if (0 == strncmp(orterun_globals.ompi_server, "file", strlen("file")) ||
0 == strncmp(orterun_globals.ompi_server, "FILE", strlen("FILE"))) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
char input[1024], *filename;
FILE *fp;
/* it is a file - get the filename */
filename = strchr(orterun_globals.ompi_server, ':');
if (NULL == filename) {
/* filename is not correctly formatted */
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:ompi-server-filename-bad", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orterun_globals.ompi_server);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
exit(1);
}
++filename; /* space past the : */
if (0 >= strlen(filename)) {
/* they forgot to give us the name! */
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:ompi-server-filename-missing", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orterun_globals.ompi_server);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
exit(1);
}
/* open the file and extract the uri */
fp = fopen(filename, "r");
if (NULL == fp) { /* can't find or read file! */
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:ompi-server-filename-access", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orterun_globals.ompi_server);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
exit(1);
}
if (NULL == fgets(input, 1024, fp)) {
/* something malformed about file */
fclose(fp);
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:ompi-server-file-bad", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orterun_globals.ompi_server,
orte_basename);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
exit(1);
}
fclose(fp);
input[strlen(input)-1] = '\0'; /* remove newline */
ompi_server = strdup(input);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
} else if (0 == strncmp(orterun_globals.ompi_server, "pid", strlen("pid")) ||
0 == strncmp(orterun_globals.ompi_server, "PID", strlen("PID"))) {
opal_list_t hnp_list;
opal_list_item_t *item;
orte_hnp_contact_t *hnp;
char *ptr;
pid_t pid;
ptr = strchr(orterun_globals.ompi_server, ':');
if (NULL == ptr) {
/* pid is not correctly formatted */
orte_show_help("help-orterun.txt", "orterun:ompi-server-pid-bad", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orte_basename,
orterun_globals.ompi_server, orte_basename);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
exit(1);
}
++ptr; /* space past the : */
if (0 >= strlen(ptr)) {
/* they forgot to give us the pid! */
orte_show_help("help-orterun.txt", "orterun:ompi-server-pid-bad", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orte_basename,
orterun_globals.ompi_server, orte_basename);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
exit(1);
}
pid = strtoul(ptr, NULL, 10);
/* to search the local mpirun's, we have to partially initialize the
- Revert r20739 This commit was SVN r20742. The following SVN revision numbers were found above: r20739 --> open-mpi/ompi@781caee0b670ea169baead8facc23390112f094f
2009-03-06 00:56:03 +03:00
* orte_process_info structure. This won't fully be setup until orte_init,
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
* but we finagle a little bit of it here
*/
- Revert r20739 This commit was SVN r20742. The following SVN revision numbers were found above: r20739 --> open-mpi/ompi@781caee0b670ea169baead8facc23390112f094f
2009-03-06 00:56:03 +03:00
if (ORTE_SUCCESS != (rc = orte_session_dir_get_name(NULL, &orte_process_info.tmpdir_base,
&orte_process_info.top_session_dir,
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
NULL, NULL, NULL))) {
orte_show_help("help-orterun.txt", "orterun:ompi-server-could-not-get-hnp-list", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orte_basename);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
exit(1);
}
OBJ_CONSTRUCT(&hnp_list, opal_list_t);
/* get the list of HNPs, but do -not- setup contact info to them in the RML */
if (ORTE_SUCCESS != (rc = orte_list_local_hnps(&hnp_list, false))) {
orte_show_help("help-orterun.txt", "orterun:ompi-server-could-not-get-hnp-list", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orte_basename);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
exit(1);
}
/* search the list for the desired pid */
while (NULL != (item = opal_list_remove_first(&hnp_list))) {
hnp = (orte_hnp_contact_t*)item;
if (pid == hnp->pid) {
ompi_server = strdup(hnp->rml_uri);
goto hnp_found;
}
OBJ_RELEASE(item);
}
/* if we got here, it wasn't found */
orte_show_help("help-orterun.txt", "orterun:ompi-server-pid-not-found", true,
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
orte_basename, orte_basename, pid, orterun_globals.ompi_server,
orte_basename);
As requested by Aurelien at the July design meeting - long time coming, but finally got around to it. Enable one mpirun to act as the server for another mpirun when doing MPI_Publish_name and its associated operations. The user is responsible, of course, for ensuring that the mpirun acting as a server outlives any mpiruns using it in that capacity. Add a cmd line option to mpirun --report-pid that prints out mpirun's pid. Allow the --ompi-server option to now take pid:# (or PID:#) of the mpirun to be used as the server, and then look that pid up by searching the local mpirun contact infos for it. This commit was SVN r20102.
2008-12-10 20:10:39 +03:00
OBJ_DESTRUCT(&hnp_list);
exit(1);
hnp_found:
/* cleanup rest of list */
while (NULL != (item = opal_list_remove_first(&hnp_list))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&hnp_list);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
} else {
ompi_server = strdup(orterun_globals.ompi_server);
}
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Make the apps */
temp_argc = 0;
temp_argv = NULL;
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_append(&temp_argc, &temp_argv, argv[0]);
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
/* NOTE: This bogus env variable is necessary in the calls to
create_app(), below. See comment immediately before the
create_app() function for an explanation. */
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
env = NULL;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
for (app_num = 0, i = 1; i < argc; ++i) {
if (0 == strcmp(argv[i], ":")) {
/* Make an app with this argv */
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
if (opal_argv_count(temp_argv) > 1) {
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
if (NULL != env) {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_free(env);
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
env = NULL;
}
Clean up a minor memory leak This commit was SVN r9413.
2006-03-24 18:28:42 +03:00
app = NULL;
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
rc = create_app(temp_argc, temp_argv, &app, &made_app, &env);
Fix the app_context problem. NOTE: JEFF SHOULD CHECK THIS! I found that orterun was not tracking the index number of the app_contexts it was creating. Hence, the app_context->idx field was always sitting at zero. This index is used by the mapper to decide which app_context to use for each process - thus, with the value of each index being zero, the mapper only used the first app_context that was created. All others were ignored. Not sure when this might have gotten changed. Could be it was a problem that always existed, but didn't get exposed until something else was changed. Anyway, it seems to work now - could stand further testing. This commit was SVN r9389.
2006-03-23 19:53:11 +03:00
/** keep track of the number of apps - point this app_context to that index */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (ORTE_SUCCESS != rc) {
/* Assume that the error message has already been
printed; no need to cleanup -- we can just
exit */
exit(1);
}
if (made_app) {
Clean up a minor memory leak This commit was SVN r9413.
2006-03-24 18:28:42 +03:00
app->idx = app_num;
++app_num;
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_add(jdata->apps, app);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
++jdata->num_apps;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Reset the temps */
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
temp_argc = 0;
temp_argv = NULL;
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_append(&temp_argc, &temp_argv, argv[0]);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
} else {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_append(&temp_argc, &temp_argv, argv[i]);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
}
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
if (opal_argv_count(temp_argv) > 1) {
Clean up a minor memory leak This commit was SVN r9413.
2006-03-24 18:28:42 +03:00
app = NULL;
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
rc = create_app(temp_argc, temp_argv, &app, &made_app, &env);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (ORTE_SUCCESS != rc) {
/* Assume that the error message has already been printed;
no need to cleanup -- we can just exit */
exit(1);
}
if (made_app) {
Clean up a minor memory leak This commit was SVN r9413.
2006-03-24 18:28:42 +03:00
app->idx = app_num;
++app_num;
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_add(jdata->apps, app);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
++jdata->num_apps;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
}
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
if (NULL != env) {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_free(env);
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
}
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_free(temp_argv);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
/* Once we've created all the apps, add the global MCA params to
each app's environment (checking for duplicates, of
course -- yay opal_environ_merge()). */
if (NULL != global_mca_env) {
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
size1 = (size_t)opal_pointer_array_get_size(jdata->apps);
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
/* Iterate through all the apps */
for (j = 0; j < size1; ++j) {
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
app = (orte_app_context_t *)
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_get_item(jdata->apps, j);
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
if (NULL != app) {
/* Use handy utility function */
env = opal_environ_merge(global_mca_env, app->env);
opal_argv_free(app->env);
app->env = env;
}
}
}
/* Now take a subset of the MCA params and set them as MCA
overrides here in orterun (so that when we orte_init() later,
all the components see these MCA params). Here's how we decide
which subset of the MCA params we set here in orterun:
1. If any global MCA params were set, use those
2. If no global MCA params were set and there was only one app,
then use its app MCA params
3. Otherwise, don't set any
*/
env = NULL;
if (NULL != global_mca_env) {
env = global_mca_env;
} else {
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
if (opal_pointer_array_get_size(jdata->apps) >= 1) {
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
/* Remember that pointer_array's can be padded with NULL
entries; so only use the app's env if there is exactly
1 non-NULL entry */
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
app = (orte_app_context_t *)
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_get_item(jdata->apps, 0);
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
if (NULL != app) {
env = app->env;
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
for (j = 1; j < opal_pointer_array_get_size(jdata->apps); ++j) {
if (NULL != opal_pointer_array_get_item(jdata->apps, j)) {
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
env = NULL;
break;
}
}
}
}
}
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
if (NULL != env) {
size1 = opal_argv_count(env);
for (j = 0; j < size1; ++j) {
Fix possible corruption of environ. Thanks to Ariel Burton and Peter Thompson for finding it! This commit was SVN r24708.
2011-05-18 20:25:35 +04:00
/* Use-after-Free error possible here. putenv does not copy
* the string passed to it, and instead stores only the pointer.
* env[j] may be freed later, in which case the pointer
* in environ will now be left dangling into a deallocated
* region.
* So we make a copy of the variable.
*/
char *s = strdup(env[j]);
if (NULL == s) {
return OPAL_ERR_OUT_OF_RESOURCE;
}
putenv(s);
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
}
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* All done */
return ORTE_SUCCESS;
}
Upgrade the ability of orterun to deal with cmd line MCA params that are passed to the orteds. Help reduce the size of the cmd line by eliminating duplicates where possible, and alert to duplicate entries that can cause problems. Add comments to both orterun and orted code explaining why we take a snapshot of the local environment and apply it to the local procs when they are spawned. This commit was SVN r18842.
2008-07-09 02:36:39 +04:00
static int capture_cmd_line_params(int argc, int start, char **argv)
{
int i, j, k;
bool ignore;
char *no_dups[] = {
"grpcomm",
"odls",
"rml",
"routed",
NULL
};
for (i = 0; i < (argc-start); ++i) {
if (0 == strcmp("-mca", argv[i]) ||
0 == strcmp("--mca", argv[i]) ) {
/* It would be nice to avoid increasing the length
* of the orted cmd line by removing any non-ORTE
* params. However, this raises a problem since
* there could be OPAL directives that we really
* -do- want the orted to see - it's only the OMPI
* related directives we could ignore. This becomes
* a very complicated procedure, however, since
* the OMPI mca params are not cleanly separated - so
* filtering them out is nearly impossible.
*
* see if this is already present so we at least can
* avoid growing the cmd line with duplicates
*/
ignore = false;
if (NULL != orted_cmd_line) {
for (j=0; NULL != orted_cmd_line[j]; j++) {
if (0 == strcmp(argv[i+1], orted_cmd_line[j])) {
/* already here - if the value is the same,
* we can quitely ignore the fact that they
* provide it more than once. However, some
* frameworks are known to have problems if the
* value is different. We don't have a good way
* to know this, but we at least make a crude
* attempt here to protect ourselves.
*/
if (0 == strcmp(argv[i+2], orted_cmd_line[j+1])) {
/* values are the same */
ignore = true;
break;
} else {
/* values are different - see if this is a problem */
for (k=0; NULL != no_dups[k]; k++) {
if (0 == strcmp(no_dups[k], argv[i+1])) {
/* print help message
* and abort as we cannot know which one is correct
*/
orte_show_help("help-orterun.txt", "orterun:conflicting-params",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
true, orte_basename, argv[i+1],
Upgrade the ability of orterun to deal with cmd line MCA params that are passed to the orteds. Help reduce the size of the cmd line by eliminating duplicates where possible, and alert to duplicate entries that can cause problems. Add comments to both orterun and orted code explaining why we take a snapshot of the local environment and apply it to the local procs when they are spawned. This commit was SVN r18842.
2008-07-09 02:36:39 +04:00
argv[i+2], orted_cmd_line[j+1]);
return ORTE_ERR_BAD_PARAM;
}
}
/* this passed muster - just ignore it */
ignore = true;
break;
}
}
}
}
if (!ignore) {
opal_argv_append_nosize(&orted_cmd_line, argv[i]);
opal_argv_append_nosize(&orted_cmd_line, argv[i+1]);
opal_argv_append_nosize(&orted_cmd_line, argv[i+2]);
}
i += 2;
}
}
return ORTE_SUCCESS;
}
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
/*
* This function takes a "char ***app_env" parameter to handle the
* specific case:
*
* orterun --mca foo bar -app appfile
*
* That is, we'll need to keep foo=bar, but the presence of the app
* file will cause an invocation of parse_appfile(), which will cause
* one or more recursive calls back to create_app(). Since the
* foo=bar value applies globally to all apps in the appfile, we need
* to pass in the "base" environment (that contains the foo=bar value)
* when we parse each line in the appfile.
*
* This is really just a special case -- when we have a simple case like:
*
* orterun --mca foo bar -np 4 hostname
*
* Then the upper-level function (parse_locals()) calls create_app()
* with a NULL value for app_env, meaning that there is no "base"
* environment that the app needs to be created from.
*/
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
static int create_app(int argc, char* argv[], orte_app_context_t **app_ptr,
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
bool *made_app, char ***app_env)
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
{
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_cmd_line_t cmd_line;
This is a very large change to rename several #define values from OMPI_* to OPAL_*. This allows opal layer to be used more independent from the whole of ompi. NOTE: 9 "svn mv" operations immediately follow this commit. This commit was SVN r21180.
2009-05-07 00:11:28 +04:00
char cwd[OPAL_PATH_MAX];
Merge in the new data support subsystem for ORTE. MPI folks should not notice a difference. Longer explanation will be sent to developers mailing list. This commit was SVN r8912.
2006-02-07 06:32:36 +03:00
int i, j, count, rc;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
char *param, *value, *value2;
orte_app_context_t *app = NULL;
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
bool cmd_line_made = false;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
*made_app = false;
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
/* Pre-process the command line if we are going to parse an appfile later.
* save any mca command line args so they can be passed
* separately to the daemons.
* Use Case:
* $ cat launch.appfile
* -np 1 -mca aaa bbb ./my-app -mca ccc ddd
* -np 1 -mca aaa bbb ./my-app -mca eee fff
* $ mpirun -np 2 -mca foo bar --app launch.appfile
* Only pick up '-mca foo bar' on this pass.
Comment out the C/N notation parsing. Interior comment has more details. This commit was SVN r7980.
2005-11-03 21:15:47 +03:00
*/
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
if (NULL != orterun_globals.appfile) {
Upgrade the ability of orterun to deal with cmd line MCA params that are passed to the orteds. Help reduce the size of the cmd line by eliminating duplicates where possible, and alert to duplicate entries that can cause problems. Add comments to both orterun and orted code explaining why we take a snapshot of the local environment and apply it to the local procs when they are spawned. This commit was SVN r18842.
2008-07-09 02:36:39 +04:00
if (ORTE_SUCCESS != (rc = capture_cmd_line_params(argc, 0, argv))) {
goto cleanup;
These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC. The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component. This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done: As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in. In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in. The incoming changes revamp these procedures in three ways: 1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step. The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic. Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure. 2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed. The size of this data has been reduced in three ways: (a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes. To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose. (b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction. (c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using. While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly. 3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup. It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging. Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future. There are a few minor additional changes in the commit that I'll just note in passing: * propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details. * requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details. * cleanup of some stale header files This commit was SVN r16364.
2007-10-05 23:48:23 +04:00
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
Upgrade the ability of orterun to deal with cmd line MCA params that are passed to the orteds. Help reduce the size of the cmd line by eliminating duplicates where possible, and alert to duplicate entries that can cause problems. Add comments to both orterun and orted code explaining why we take a snapshot of the local environment and apply it to the local procs when they are spawned. This commit was SVN r18842.
2008-07-09 02:36:39 +04:00
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
/* Parse application command line options. */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
init_globals();
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_cmd_line_create(&cmd_line, cmd_line_init);
* Merge changes from tim branch from r 4821 to 4892. Tree can now run MPI and non-ORTE applications for RSH on one node with or without threads. I think we're approaching convergence with the tim branch This commit was SVN r4895.
2005-03-18 06:43:59 +03:00
mca_base_cmd_line_setup(&cmd_line);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
cmd_line_made = true;
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
rc = opal_cmd_line_parse(&cmd_line, true, argc, argv);
Next step in the project split, mainly source code re-arranging - move files out of toplevel include/ and etc/, moving it into the sub-projects - rather than including config headers with <project>/include, have them as <project> - require all headers to be included with a project prefix, with the exception of the config headers ({opal,orte,ompi}_config.h mpi.h, and mpif.h) This commit was SVN r8985.
2006-02-12 04:33:29 +03:00
if (ORTE_SUCCESS != rc) {
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
goto cleanup;
}
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
mca_base_cmd_line_process_args(&cmd_line, app_env, &global_mca_env);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Is there an appfile in here? */
if (NULL != orterun_globals.appfile) {
OBJ_DESTRUCT(&cmd_line);
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
return parse_appfile(strdup(orterun_globals.appfile), app_env);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
/* Setup application context */
app = OBJ_NEW(orte_app_context_t);
Merge in the new data support subsystem for ORTE. MPI folks should not notice a difference. Longer explanation will be sent to developers mailing list. This commit was SVN r8912.
2006-02-07 06:32:36 +03:00
opal_cmd_line_get_tail(&cmd_line, &count, &app->argv);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* See if we have anything left */
Merge in the new data support subsystem for ORTE. MPI folks should not notice a difference. Longer explanation will be sent to developers mailing list. This commit was SVN r8912.
2006-02-07 06:32:36 +03:00
if (0 == count) {
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:executable-not-specified",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
true, orte_basename, orte_basename);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
rc = ORTE_ERR_NOT_FOUND;
goto cleanup;
}
Fix orterun so it does not get influenced by an application's argv set. For example, if I have an application that, internal to the application, takes the argument '-mca foo bar' we do not want orterun to pick up this argument and pass it through the system. So the following {{{ shell$ mpirun -np 2 -mca btl tcp,self ./myapp -mca foo bar }}} orterun should pick up {{{-mca btl tcp,self}}} but not {{{-mca foo bar}}} which it was previous to this commit. I tested command line runs and runs with app files to confirm this patch works. This commit was SVN r16431.
2007-10-11 22:33:40 +04:00
/*
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
* Get mca parameters so we can pass them to the daemons.
* Use the count determined above to make sure we do not go past
* the executable name. Example:
Fix orterun so it does not get influenced by an application's argv set. For example, if I have an application that, internal to the application, takes the argument '-mca foo bar' we do not want orterun to pick up this argument and pass it through the system. So the following {{{ shell$ mpirun -np 2 -mca btl tcp,self ./myapp -mca foo bar }}} orterun should pick up {{{-mca btl tcp,self}}} but not {{{-mca foo bar}}} which it was previous to this commit. I tested command line runs and runs with app files to confirm this patch works. This commit was SVN r16431.
2007-10-11 22:33:40 +04:00
* mpirun -np 2 -mca foo bar ./my-app -mca bip bop
* We want to pick up '-mca foo bar' but not '-mca bip bop'
*/
Upgrade the ability of orterun to deal with cmd line MCA params that are passed to the orteds. Help reduce the size of the cmd line by eliminating duplicates where possible, and alert to duplicate entries that can cause problems. Add comments to both orterun and orted code explaining why we take a snapshot of the local environment and apply it to the local procs when they are spawned. This commit was SVN r18842.
2008-07-09 02:36:39 +04:00
if (ORTE_SUCCESS != (rc = capture_cmd_line_params(argc, count, argv))) {
goto cleanup;
Fix orterun so it does not get influenced by an application's argv set. For example, if I have an application that, internal to the application, takes the argument '-mca foo bar' we do not want orterun to pick up this argument and pass it through the system. So the following {{{ shell$ mpirun -np 2 -mca btl tcp,self ./myapp -mca foo bar }}} orterun should pick up {{{-mca btl tcp,self}}} but not {{{-mca foo bar}}} which it was previous to this commit. I tested command line runs and runs with app files to confirm this patch works. This commit was SVN r16431.
2007-10-11 22:33:40 +04:00
}
Upgrade the ability of orterun to deal with cmd line MCA params that are passed to the orteds. Help reduce the size of the cmd line by eliminating duplicates where possible, and alert to duplicate entries that can cause problems. Add comments to both orterun and orted code explaining why we take a snapshot of the local environment and apply it to the local procs when they are spawned. This commit was SVN r18842.
2008-07-09 02:36:39 +04:00
Fix two thing d about orterun that have been bugging me for a little bit: - when you hit ctrl-c, say "argv[0]: aborting..." -- don't hard-wire it to "orterun: aborting..." - if you don't specify -np or a mapping arugment, barf before trying to rmgr_spawn and giving an unintelligible error message about being out of resources (becase we try to allocate -1 nodes). This commit was SVN r5234.
2005-04-09 05:26:17 +04:00
/* Grab all OMPI_* environment variables */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
app->env = opal_argv_copy(*app_env);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
for (i = 0; NULL != environ[i]; ++i) {
Export any environment variable that starts with OMPI_* not just those that start with OMPI_MCA_* This commit was SVN r5190.
2005-04-06 05:58:30 +04:00
if (0 == strncmp("OMPI_", environ[i], 5)) {
Properly ensure that cmd line MCA params override environmental MCA params for procs local to mpirun. Actually, the problem was that we were simply -adding- any enviro MCA params to whatever had been found on the cmd line. Thus, duplicate MCA param directives were winding up duplicated in the environment. Some shells took the first one in the environ array - others took the last! So we could get completely different behavior based on the whims of the shell. This commit fixes trac:1373 This commit was SVN r18836. The following Trac tickets were found above: Ticket 1373 --> https://svn.open-mpi.org/trac/ompi/ticket/1373
2008-07-08 17:48:47 +04:00
/* check for duplicate in app->env - this
* would have been placed there by the
* cmd line processor. By convention, we
* always let the cmd line override the
* environment
*/
param = strdup(environ[i]);
value = strchr(param, '=');
*value = '\0';
value++;
opal_setenv(param, value, false, &app->env);
free(param);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
}
Enable modex-less launch. Consists of: 1. minor modification to include two new opal MCA params: (a) opal_profile: outputs what components were selected by each framework currently enabled for most, but not all, frameworks (b) opal_profile_file: name of file that contains profile info required for modex 2. introduction of two new tools: (a) ompi-probe: MPI process that simply calls MPI_Init/Finalize with opal_profile set. Also reports back the rml IP address for all interfaces on the node (b) ompi-profiler: uses ompi-probe to create the profile_file, also reports out a summary of what framework components are actually being used to help with configuration options 3. modification of the grpcomm basic component to utilize the profile file in place of the modex where possible 4. modification of orterun so it properly sees opal mca params and handles opal_profile correctly to ensure we don't get its profile 5. similar mod to orted as for orterun 6. addition of new test that calls orte_init followed by calls to grpcomm.barrier This is all completely benign unless actively selected. At the moment, it only supports modex-less launch for openib-based systems. Minor mod to the TCP btl would be required to enable it as well, if people are interested. Similarly, anyone interested in enabling other BTL's for modex-less operation should let me know and I'll give you the magic details. This seems to significantly improve scalability provided the file can be locally located on the nodes. I'm looking at an alternative means of disseminating the info (perhaps in launch message) as an option for removing that constraint. This commit was SVN r20098.
2008-12-10 02:49:02 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* add the ompi-server, if provided */
if (NULL != ompi_server) {
Enable modex-less launch. Consists of: 1. minor modification to include two new opal MCA params: (a) opal_profile: outputs what components were selected by each framework currently enabled for most, but not all, frameworks (b) opal_profile_file: name of file that contains profile info required for modex 2. introduction of two new tools: (a) ompi-probe: MPI process that simply calls MPI_Init/Finalize with opal_profile set. Also reports back the rml IP address for all interfaces on the node (b) ompi-profiler: uses ompi-probe to create the profile_file, also reports out a summary of what framework components are actually being used to help with configuration options 3. modification of the grpcomm basic component to utilize the profile file in place of the modex where possible 4. modification of orterun so it properly sees opal mca params and handles opal_profile correctly to ensure we don't get its profile 5. similar mod to orted as for orterun 6. addition of new test that calls orte_init followed by calls to grpcomm.barrier This is all completely benign unless actively selected. At the moment, it only supports modex-less launch for openib-based systems. Minor mod to the TCP btl would be required to enable it as well, if people are interested. Similarly, anyone interested in enabling other BTL's for modex-less operation should let me know and I'll give you the magic details. This seems to significantly improve scalability provided the file can be locally located on the nodes. I'm looking at an alternative means of disseminating the info (perhaps in launch message) as an option for removing that constraint. This commit was SVN r20098.
2008-12-10 02:49:02 +03:00
opal_setenv("OMPI_MCA_pubsub_orte_server", ompi_server, true, &app->env);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
}
Remove stale orte_vm_launch param. Add a param that allows users to specify envars to forward/set so they can do it in the MCA param file instead of only via mpirun cmd line. This commit was SVN r25580.
2011-12-07 01:31:22 +04:00
/* Did the user request to export any environment variables on the cmd line? */
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
if (opal_cmd_line_is_taken(&cmd_line, "x")) {
j = opal_cmd_line_get_ninsts(&cmd_line, "x");
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
for (i = 0; i < j; ++i) {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
param = opal_cmd_line_get_param(&cmd_line, "x", i, 0);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (NULL != strchr(param, '=')) {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_append_nosize(&app->env, param);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
} else {
value = getenv(param);
if (NULL != value) {
if (NULL != strchr(value, '=')) {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_append_nosize(&app->env, value);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
} else {
asprintf(&value2, "%s=%s", param, value);
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_append_nosize(&app->env, value2);
Bring in the remote launch changes. This still isn't fully functional, but impacted a few other places that were worth fixing. 1. Added a new function to launch head node processes on remote nodes. 2. Added new tool "orteprobe" that checks to see if a daemon is running on a node. If so, it reports the contact info back to the requestor. If not, it will (eventually - but not now) fork/exec a daemon on the node, report the contact info back to requestor, and then die. 3. Modified orted to handle universe name parameters, and added separate command line flags for debugging the daemon and saving daemon debugging output in a file. The "debug" flag now turns on the runtime debug info instead of the daemon debug - thus, you can now just get daemon debug info if you like. 4. Fix the dps to handle zero length strings correctly. 5. Modify the fork and rsh launchers to pass required environmental variables to the daemons and processes 6. Pulled the redirection of stdin/stdout/stderr for the daemon out of orted and put it into the daemon_init function to simplify orted logic. 7. Modified sys_info to correctly deal with passed mca param 8. Modified univ_info to parse incoming universe location information. This commit was SVN r5705.
2005-05-13 01:44:23 +04:00
free(value2);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
} else {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output(0, "Warning: could not find environment variable \"%s\"\n", param);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
}
}
}
Remove stale orte_vm_launch param. Add a param that allows users to specify envars to forward/set so they can do it in the MCA param file instead of only via mpirun cmd line. This commit was SVN r25580.
2011-12-07 01:31:22 +04:00
/* Did the user request to export any environment variables via MCA param? */
if (NULL != orte_forward_envars) {
char **vars;
vars = opal_argv_split(orte_forward_envars, ',');
for (i=0; NULL != vars[i]; i++) {
if (NULL != strchr(vars[i], '=')) {
/* user supplied a value */
opal_argv_append_nosize(&app->env, vars[i]);
} else {
/* get the value from the environ */
value = getenv(vars[i]);
if (NULL != value) {
if (NULL != strchr(value, '=')) {
opal_argv_append_nosize(&app->env, value);
} else {
asprintf(&value2, "%s=%s", vars[i], value);
opal_argv_append_nosize(&app->env, value2);
free(value2);
}
} else {
opal_output(0, "Warning: could not find environment variable \"%s\"\n", param);
}
}
}
opal_argv_free(vars);
}
Fix the use of --path and -x PATH so that things work properly. Note that --path specifies extra directories where the executable is searched for, but does not affect the PATH settings. This commit fixes trac:1221. This commit was SVN r17748. The following Trac tickets were found above: Ticket 1221 --> https://svn.open-mpi.org/trac/ompi/ticket/1221
2008-03-06 00:07:43 +03:00
/* If the user specified --path, store it in the user's app
environment via the OMPI_exec_path variable. */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (NULL != orterun_globals.path) {
Fix the use of --path and -x PATH so that things work properly. Note that --path specifies extra directories where the executable is searched for, but does not affect the PATH settings. This commit fixes trac:1221. This commit was SVN r17748. The following Trac tickets were found above: Ticket 1221 --> https://svn.open-mpi.org/trac/ompi/ticket/1221
2008-03-06 00:07:43 +03:00
asprintf(&value, "OMPI_exec_path=%s", orterun_globals.path);
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_append_nosize(&app->env, value);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
free(value);
}
/* Did the user request a specific wdir? */
if (NULL != orterun_globals.wdir) {
Improve the efficiency of the launch system by changing the outer loop to being over app_context, and adding a flag to the app_context so the daemon can record that "this app is on my node" when decoding the launch msg. If the --wdir option is given, check to see if the user provided a relative path. If so, convert it to an absolute path. This is needed to maintain consistent behavior across environements. Some environments automatically chdir to your current working directory when launching the remote orted, while others (e.g., ssh) don't. This levels the playing field and reduces user surprise. This commit was SVN r20342.
2009-01-25 15:39:24 +03:00
/* if this is a relative path, convert it to an absolute path */
if (opal_path_is_absolute(orterun_globals.wdir)) {
app->cwd = strdup(orterun_globals.wdir);
} else {
/* get the cwd */
if (OPAL_SUCCESS != (rc = opal_getcwd(cwd, sizeof(cwd)))) {
orte_show_help("help-orterun.txt", "orterun:init-failure",
true, "get the cwd", rc);
goto cleanup;
}
/* construct the absolute path */
app->cwd = opal_os_path(false, cwd, orterun_globals.wdir, NULL);
}
Change the behavior of orterun (mpirun, mpirexec) to search for argv[0] and the cwd on the target node (i.e., the node where the executable will be running in all systems except BProc, where the searches are run on the node where orterun is invoked). - fork pls now does cwd and argv[0] search in orted - bproc pls does cwd and argv[0] search in orterun - cwd behavior slightly different: - if user specifies a -wdir to orterun, we chdir() to there; if we can't for some reason, abort - if user does not specify a -wdir, try to chdir() to the dir where orterun was invoked. If we can't for some reason (e.g., it doesn't exist on the target node), then try to chdir($HOME). If we can't do that, then just live with whatever default directory we were put in. This commit was SVN r9068.
2006-02-16 23:40:23 +03:00
app->user_specified_cwd = true;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
} else {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
if (OPAL_SUCCESS != (rc = opal_getcwd(cwd, sizeof(cwd)))) {
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:init-failure",
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
true, "get the cwd", rc);
goto cleanup;
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
app->cwd = strdup(cwd);
Change the behavior of orterun (mpirun, mpirexec) to search for argv[0] and the cwd on the target node (i.e., the node where the executable will be running in all systems except BProc, where the searches are run on the node where orterun is invoked). - fork pls now does cwd and argv[0] search in orted - bproc pls does cwd and argv[0] search in orterun - cwd behavior slightly different: - if user specifies a -wdir to orterun, we chdir() to there; if we can't for some reason, abort - if user does not specify a -wdir, try to chdir() to the dir where orterun was invoked. If we can't for some reason (e.g., it doesn't exist on the target node), then try to chdir($HOME). If we can't do that, then just live with whatever default directory we were put in. This commit was SVN r9068.
2006-02-16 23:40:23 +03:00
app->user_specified_cwd = false;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
Fixes trac:377 Add --enable-orterun-prefix-by-default (and a synonym: --enable-mpirun-prefix-by-default) to make orterun always behave as if "--prefix $prefix" was given on the command line (where $prefix is the value given to the --prefix option to configure). This prevents many rsh/ssh users from needing to modify their shell startup files to set the LD_LIBRARY_PATH for Open MPI (they will still need to set PATH or otherwise find the OMPI executables to mpicc/mpirun/etc. their MPI applications). Also added --noprefix option to orterun to disable this behavior. Finally, note that even if --enable-orterun-prefix-by-default is specified, if the user specifies --prefix or /path/to/mpirun, these options will override the default value of the prefix ($prefix). This commit was SVN r11669. The following Trac tickets were found above: Ticket 377 --> https://svn.open-mpi.org/trac/ompi/ticket/377
2006-09-15 06:52:08 +04:00
/* Check to see if the user explicitly wanted to disable automatic
--prefix behavior */
if (opal_cmd_line_is_taken(&cmd_line, "noprefix")) {
want_prefix_by_default = false;
}
- Fix bug in orterun where --prefix didn't show up in the help output (reported by Cisco) - While in orterun, add a feature that multiple users have asked for: if you specify an absolute pathname to orterun, such as "/path/to/bin/orterun ...", it's equivalent to "orterun --path /path/to ..." This commit was SVN r9181.
2006-02-28 14:52:12 +03:00
/* Did the user specify a specific prefix for this app_context_t
or provide an absolute path name to argv[0]? */
if (opal_cmd_line_is_taken(&cmd_line, "prefix") ||
Fixes trac:377 Add --enable-orterun-prefix-by-default (and a synonym: --enable-mpirun-prefix-by-default) to make orterun always behave as if "--prefix $prefix" was given on the command line (where $prefix is the value given to the --prefix option to configure). This prevents many rsh/ssh users from needing to modify their shell startup files to set the LD_LIBRARY_PATH for Open MPI (they will still need to set PATH or otherwise find the OMPI executables to mpicc/mpirun/etc. their MPI applications). Also added --noprefix option to orterun to disable this behavior. Finally, note that even if --enable-orterun-prefix-by-default is specified, if the user specifies --prefix or /path/to/mpirun, these options will override the default value of the prefix ($prefix). This commit was SVN r11669. The following Trac tickets were found above: Ticket 377 --> https://svn.open-mpi.org/trac/ompi/ticket/377
2006-09-15 06:52:08 +04:00
'/' == argv[0][0] || want_prefix_by_default) {
- Support -prefix specification on mpirun/orterun cmd-line per app_context: mpirun -np 2 -prefix /path/to/ompi/on/machineA ./exec1 : \ -np 2 -prefix /path/to/ompi/on/machineB ./exec2 - Allow with -mca pls_rsh_assume_same_shell 0, the checking for the SHELL-variable on the actual node (currently 1st node). Sets the prefix, PATH and LD_LIBRARY_PATH for bash/ksh and csh/tcsh. This commit was SVN r7195.
2005-09-06 20:10:05 +04:00
size_t param_len;
Grr...get the prefix right - need to strip the bin out of absolute path to mpirun. This commit was SVN r24658.
2011-04-29 02:20:55 +04:00
char *path_to_mpirun=NULL;
- Support -prefix specification on mpirun/orterun cmd-line per app_context: mpirun -np 2 -prefix /path/to/ompi/on/machineA ./exec1 : \ -np 2 -prefix /path/to/ompi/on/machineB ./exec2 - Allow with -mca pls_rsh_assume_same_shell 0, the checking for the SHELL-variable on the actual node (currently 1st node). Sets the prefix, PATH and LD_LIBRARY_PATH for bash/ksh and csh/tcsh. This commit was SVN r7195.
2005-09-06 20:10:05 +04:00
Grr...get the prefix right - need to strip the bin out of absolute path to mpirun. This commit was SVN r24658.
2011-04-29 02:20:55 +04:00
if ('/' == argv[0][0]) {
char* tmp_basename = NULL;
/* If they specified an absolute path, strip off the
/bin/<exec_name>" and leave just the prefix */
path_to_mpirun = opal_dirname(argv[0]);
/* Quick sanity check to ensure we got
something/bin/<exec_name> and that the installation
tree is at least more or less what we expect it to
be */
tmp_basename = opal_basename(path_to_mpirun);
if (0 == strcmp("bin", tmp_basename)) {
char* tmp = path_to_mpirun;
path_to_mpirun = opal_dirname(tmp);
free(tmp);
} else {
free(path_to_mpirun);
path_to_mpirun = NULL;
}
free(tmp_basename);
}
Check for both absolute-path-to-mpirun and -prefix being specified. If the two differ, print out a warning and ignore -prefix. If they are the same, or only one was given, then proceed as directed. This commit was SVN r24657.
2011-04-29 02:12:41 +04:00
/* if both are given, check to see if they match */
Grr...get the prefix right - need to strip the bin out of absolute path to mpirun. This commit was SVN r24658.
2011-04-29 02:20:55 +04:00
if (opal_cmd_line_is_taken(&cmd_line, "prefix") && NULL != path_to_mpirun) {
Check for both absolute-path-to-mpirun and -prefix being specified. If the two differ, print out a warning and ignore -prefix. If they are the same, or only one was given, then proceed as directed. This commit was SVN r24657.
2011-04-29 02:12:41 +04:00
/* if they don't match, then that merits a warning */
Grr...get the prefix right - need to strip the bin out of absolute path to mpirun. This commit was SVN r24658.
2011-04-29 02:20:55 +04:00
param = strdup(opal_cmd_line_get_param(&cmd_line, "prefix", 0, 0));
Check for both absolute-path-to-mpirun and -prefix being specified. If the two differ, print out a warning and ignore -prefix. If they are the same, or only one was given, then proceed as directed. This commit was SVN r24657.
2011-04-29 02:12:41 +04:00
if (0 != strcmp(param, path_to_mpirun)) {
orte_show_help("help-orterun.txt", "orterun:double-prefix",
true, orte_basename, orte_basename, param, path_to_mpirun);
/* let the path-to-mpirun take precedence since we
* know that one is being used
*/
Grr...get the prefix right - need to strip the bin out of absolute path to mpirun. This commit was SVN r24658.
2011-04-29 02:20:55 +04:00
free(param);
Check for both absolute-path-to-mpirun and -prefix being specified. If the two differ, print out a warning and ignore -prefix. If they are the same, or only one was given, then proceed as directed. This commit was SVN r24657.
2011-04-29 02:12:41 +04:00
param = path_to_mpirun;
} else {
/* since they match, just use param */
free(path_to_mpirun);
}
Grr...get the prefix right - need to strip the bin out of absolute path to mpirun. This commit was SVN r24658.
2011-04-29 02:20:55 +04:00
} else if (NULL != path_to_mpirun) {
param = path_to_mpirun;
Check for both absolute-path-to-mpirun and -prefix being specified. If the two differ, print out a warning and ignore -prefix. If they are the same, or only one was given, then proceed as directed. This commit was SVN r24657.
2011-04-29 02:12:41 +04:00
} else if (opal_cmd_line_is_taken(&cmd_line, "prefix")){
/* must be --prefix alone */
param = strdup(opal_cmd_line_get_param(&cmd_line, "prefix", 0, 0));
} else {
/* --enable-orterun-prefix-default was given to orterun */
param = strdup(opal_install_dirs.prefix);
Fixes trac:377 Add --enable-orterun-prefix-by-default (and a synonym: --enable-mpirun-prefix-by-default) to make orterun always behave as if "--prefix $prefix" was given on the command line (where $prefix is the value given to the --prefix option to configure). This prevents many rsh/ssh users from needing to modify their shell startup files to set the LD_LIBRARY_PATH for Open MPI (they will still need to set PATH or otherwise find the OMPI executables to mpicc/mpirun/etc. their MPI applications). Also added --noprefix option to orterun to disable this behavior. Finally, note that even if --enable-orterun-prefix-by-default is specified, if the user specifies --prefix or /path/to/mpirun, these options will override the default value of the prefix ($prefix). This commit was SVN r11669. The following Trac tickets were found above: Ticket 377 --> https://svn.open-mpi.org/trac/ompi/ticket/377
2006-09-15 06:52:08 +04:00
}
- Support -prefix specification on mpirun/orterun cmd-line per app_context: mpirun -np 2 -prefix /path/to/ompi/on/machineA ./exec1 : \ -np 2 -prefix /path/to/ompi/on/machineB ./exec2 - Allow with -mca pls_rsh_assume_same_shell 0, the checking for the SHELL-variable on the actual node (currently 1st node). Sets the prefix, PATH and LD_LIBRARY_PATH for bash/ksh and csh/tcsh. This commit was SVN r7195.
2005-09-06 20:10:05 +04:00
- Fix bug in orterun where --prefix didn't show up in the help output (reported by Cisco) - While in orterun, add a feature that multiple users have asked for: if you specify an absolute pathname to orterun, such as "/path/to/bin/orterun ...", it's equivalent to "orterun --path /path/to ..." This commit was SVN r9181.
2006-02-28 14:52:12 +03:00
if (NULL != param) {
Correct a comment and protect the usage of the environ variable against Windows. This commit was SVN r11397.
2006-08-24 20:18:42 +04:00
/* "Parse" the param, aka remove superfluous path_sep. */
- Fix bug in orterun where --prefix didn't show up in the help output (reported by Cisco) - While in orterun, add a feature that multiple users have asked for: if you specify an absolute pathname to orterun, such as "/path/to/bin/orterun ...", it's equivalent to "orterun --path /path/to ..." This commit was SVN r9181.
2006-02-28 14:52:12 +03:00
param_len = strlen(param);
The names of the defines should start with OPAL as they belong to the OPAL layer. We now support 64 bits Windows too. This commit was SVN r11312.
2006-08-22 01:55:41 +04:00
while (0 == strcmp (OPAL_PATH_SEP, &(param[param_len-1]))) {
- Fix bug in orterun where --prefix didn't show up in the help output (reported by Cisco) - While in orterun, add a feature that multiple users have asked for: if you specify an absolute pathname to orterun, such as "/path/to/bin/orterun ...", it's equivalent to "orterun --path /path/to ..." This commit was SVN r9181.
2006-02-28 14:52:12 +03:00
param[param_len-1] = '\0';
param_len--;
if (0 == param_len) {
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:empty-prefix",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
true, orte_basename, orte_basename);
- Fix bug in orterun where --prefix didn't show up in the help output (reported by Cisco) - While in orterun, add a feature that multiple users have asked for: if you specify an absolute pathname to orterun, such as "/path/to/bin/orterun ...", it's equivalent to "orterun --path /path/to ..." This commit was SVN r9181.
2006-02-28 14:52:12 +03:00
return ORTE_ERR_FATAL;
}
}
app->prefix_dir = strdup(param);
Check for both absolute-path-to-mpirun and -prefix being specified. If the two differ, print out a warning and ignore -prefix. If they are the same, or only one was given, then proceed as directed. This commit was SVN r24657.
2011-04-29 02:12:41 +04:00
free(param);
- Fix bug in orterun where --prefix didn't show up in the help output (reported by Cisco) - While in orterun, add a feature that multiple users have asked for: if you specify an absolute pathname to orterun, such as "/path/to/bin/orterun ...", it's equivalent to "orterun --path /path/to ..." This commit was SVN r9181.
2006-02-28 14:52:12 +03:00
}
- Support -prefix specification on mpirun/orterun cmd-line per app_context: mpirun -np 2 -prefix /path/to/ompi/on/machineA ./exec1 : \ -np 2 -prefix /path/to/ompi/on/machineB ./exec2 - Allow with -mca pls_rsh_assume_same_shell 0, the checking for the SHELL-variable on the actual node (currently 1st node). Sets the prefix, PATH and LD_LIBRARY_PATH for bash/ksh and csh/tcsh. This commit was SVN r7195.
2005-09-06 20:10:05 +04:00
}
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
/* Did the user specify a hostfile. Need to check for both
* hostfile and machine file.
* We can only deal with one hostfile per app context, otherwise give an error.
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
*/
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
if (0 < (j = opal_cmd_line_get_ninsts(&cmd_line, "hostfile"))) {
if(1 < j) {
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:multiple-hostfiles",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
true, orte_basename, NULL);
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
return ORTE_ERR_FATAL;
} else {
value = opal_cmd_line_get_param(&cmd_line, "hostfile", 0, 0);
app->hostfile = strdup(value);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
}
if (0 < (j = opal_cmd_line_get_ninsts(&cmd_line, "machinefile"))) {
if(1 < j || NULL != app->hostfile) {
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:multiple-hostfiles",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
true, orte_basename, NULL);
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
return ORTE_ERR_FATAL;
} else {
value = opal_cmd_line_get_param(&cmd_line, "machinefile", 0, 0);
app->hostfile = strdup(value);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
}
/* Did the user specify any hosts? */
if (0 < (j = opal_cmd_line_get_ninsts(&cmd_line, "host"))) {
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
for (i = 0; i < j; ++i) {
Make it so we do not mangle the options the user passes to their executeable. Fixes trac:1124 The change also: - cleans up and simplifies the command line processing code - adds an error output if more than one hostfile passed for a single app context - gets rid of the superfluous orte_app_context_map_t type, and instead use a simple argv of -host options This commit was SVN r17750. The following Trac tickets were found above: Ticket 1124 --> https://svn.open-mpi.org/trac/ompi/ticket/1124
2008-03-06 01:12:27 +03:00
value = opal_cmd_line_get_param(&cmd_line, "host", i, 0);
opal_argv_append_nosize(&app->dash_host, value);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
}
/* Get the numprocs */
Fix for #397 on 64 bit platforms sizeof(size_t) != sizeof(orte_std_cntr_t), and we were incorrectly assuming this when dealing with num procs. It worked on little endian platforms, but not big endian. So change num_procs to type int, and cast where needed. This commit was SVN r11796.
2006-09-25 23:41:54 +04:00
app->num_procs = (orte_std_cntr_t)orterun_globals.num_procs;
Fix two thing d about orterun that have been bugging me for a little bit: - when you hit ctrl-c, say "argv[0]: aborting..." -- don't hard-wire it to "orterun: aborting..." - if you don't specify -np or a mapping arugment, barf before trying to rmgr_spawn and giving an unintelligible error message about being out of resources (becase we try to allocate -1 nodes). This commit was SVN r5234.
2005-04-09 05:26:17 +04:00
Per LANL's stated need, add functionality that runs a.out across ALL available process slots if no num_proc is specified on the command line. However, please note the following limitation: we ONLY allow ONE application to be specified on the command line when this feature is invoked. If multiple apps are specified, the user MUST also specify the number to be launched for each and every one of them. Update the help text to report errors when not following that rule. Also updated the RMAPS help text to reflect the reorganization of some of the round-robin code into the base. The new functionality has been tested under Mac OS-X and on Odin using an MPI program. Both byslot and bynode mapping have been checked and verified. Operational support for other systems needs to be verified - I respectfully request people's help in doing so. This commit was SVN r10708.
2006-07-11 01:25:33 +04:00
total_num_apps++;
Merging in the jjhursey-ft-cr-stable branch (r13912 : HEAD). This merge adds Checkpoint/Restart support to Open MPI. The initial frameworks and components support a LAM/MPI-like implementation. This commit follows the risk assessment presented to the Open MPI core development group on Feb. 22, 2007. This commit closes trac:158 More details to follow. This commit was SVN r14051. The following SVN revisions from the original message are invalid or inconsistent and therefore were not cross-referenced: r13912 The following Trac tickets were found above: Ticket 158 --> https://svn.open-mpi.org/trac/ompi/ticket/158
2007-03-17 02:11:45 +03:00
/* Preserve if we are to preload the binary */
app->preload_binary = orterun_globals.preload_binary;
if( NULL != orterun_globals.preload_files)
app->preload_files = strdup(orterun_globals.preload_files);
else
app->preload_files = NULL;
if( NULL != orterun_globals.preload_files_dest_dir)
app->preload_files_dest_dir = strdup(orterun_globals.preload_files_dest_dir);
else
app->preload_files_dest_dir = NULL;
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_FT_CR == 1
if( NULL != orterun_globals.sstore_load ) {
app->sstore_load = strdup(orterun_globals.sstore_load);
} else {
app->sstore_load = NULL;
}
#endif
Merging in the jjhursey-ft-cr-stable branch (r13912 : HEAD). This merge adds Checkpoint/Restart support to Open MPI. The initial frameworks and components support a LAM/MPI-like implementation. This commit follows the risk assessment presented to the Open MPI core development group on Feb. 22, 2007. This commit closes trac:158 More details to follow. This commit was SVN r14051. The following SVN revisions from the original message are invalid or inconsistent and therefore were not cross-referenced: r13912 The following Trac tickets were found above: Ticket 158 --> https://svn.open-mpi.org/trac/ompi/ticket/158
2007-03-17 02:11:45 +03:00
Change the behavior of orterun (mpirun, mpirexec) to search for argv[0] and the cwd on the target node (i.e., the node where the executable will be running in all systems except BProc, where the searches are run on the node where orterun is invoked). - fork pls now does cwd and argv[0] search in orted - bproc pls does cwd and argv[0] search in orterun - cwd behavior slightly different: - if user specifies a -wdir to orterun, we chdir() to there; if we can't for some reason, abort - if user does not specify a -wdir, try to chdir() to the dir where orterun was invoked. If we can't for some reason (e.g., it doesn't exist on the target node), then try to chdir($HOME). If we can't do that, then just live with whatever default directory we were put in. This commit was SVN r9068.
2006-02-16 23:40:23 +03:00
/* Do not try to find argv[0] here -- the starter is responsible
for that because it may not be relevant to try to find it on
the node where orterun is executing. So just strdup() argv[0]
into app. */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Change the behavior of orterun (mpirun, mpirexec) to search for argv[0] and the cwd on the target node (i.e., the node where the executable will be running in all systems except BProc, where the searches are run on the node where orterun is invoked). - fork pls now does cwd and argv[0] search in orted - bproc pls does cwd and argv[0] search in orterun - cwd behavior slightly different: - if user specifies a -wdir to orterun, we chdir() to there; if we can't for some reason, abort - if user does not specify a -wdir, try to chdir() to the dir where orterun was invoked. If we can't for some reason (e.g., it doesn't exist on the target node), then try to chdir($HOME). If we can't do that, then just live with whatever default directory we were put in. This commit was SVN r9068.
2006-02-16 23:40:23 +03:00
app->app = strdup(app->argv[0]);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (NULL == app->app) {
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:call-failed",
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
true, orte_basename, "library", "strdup returned NULL", errno);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
rc = ORTE_ERR_NOT_FOUND;
goto cleanup;
}
*app_ptr = app;
app = NULL;
*made_app = true;
/* All done */
cleanup:
if (NULL != app) {
OBJ_RELEASE(app);
}
if (cmd_line_made) {
OBJ_DESTRUCT(&cmd_line);
}
return rc;
}
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
static int parse_appfile(char *filename, char ***env)
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
{
size_t i, len;
FILE *fp;
char line[BUFSIZ];
Fix the app_context problem for app_files too.... Again, this should be checked by Jeff. This commit was SVN r9393.
2006-03-23 20:55:25 +03:00
int rc, argc, app_num;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
char **argv;
orte_app_context_t *app;
bool blank, made_app;
char bogus[] = "bogus ";
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
char **tmp_env;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
Fix orterun so it does not get influenced by an application's argv set. For example, if I have an application that, internal to the application, takes the argument '-mca foo bar' we do not want orterun to pick up this argument and pass it through the system. So the following {{{ shell$ mpirun -np 2 -mca btl tcp,self ./myapp -mca foo bar }}} orterun should pick up {{{-mca btl tcp,self}}} but not {{{-mca foo bar}}} which it was previous to this commit. I tested command line runs and runs with app files to confirm this patch works. This commit was SVN r16431.
2007-10-11 22:33:40 +04:00
/*
* Make sure to clear out this variable so we don't do anything odd in
* app_create()
*/
if( NULL != orterun_globals.appfile ) {
free( orterun_globals.appfile );
orterun_globals.appfile = NULL;
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
/* Try to open the file */
fp = fopen(filename, "r");
if (NULL == fp) {
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
orte_show_help("help-orterun.txt", "orterun:appfile-not-found", true,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
filename);
return ORTE_ERR_NOT_FOUND;
}
/* Read in line by line */
line[sizeof(line) - 1] = '\0';
Fix the app_context problem for app_files too.... Again, this should be checked by Jeff. This commit was SVN r9393.
2006-03-23 20:55:25 +03:00
app_num = 0;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
do {
/* We need a bogus argv[0] (because when argv comes in from
the command line, argv[0] is "orterun", so the parsing
logic ignores it). So create one here rather than making
an argv and then pre-pending a new argv[0] (which would be
rather inefficient). */
line[0] = '\0';
strcat(line, bogus);
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
if (NULL == fgets(line + sizeof(bogus) - 1,
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
sizeof(line) - sizeof(bogus) - 1, fp)) {
break;
}
Ensure to remove trailing newlines in the command line when parsing appfiles This commit was SVN r5271.
2005-04-12 22:42:34 +04:00
/* Remove a trailing newline */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
len = strlen(line);
Ensure to remove trailing newlines in the command line when parsing appfiles This commit was SVN r5271.
2005-04-12 22:42:34 +04:00
if (len > 0 && '\n' == line[len - 1]) {
line[len - 1] = '\0';
if (len > 0) {
--len;
}
}
/* Remove comments */
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
for (i = 0; i < len; ++i) {
if ('#' == line[i]) {
line[i] = '\0';
break;
} else if (i + 1 < len && '/' == line[i] && '/' == line[i + 1]) {
line[i] = '\0';
break;
}
}
/* Is this a blank line? */
len = strlen(line);
for (blank = true, i = sizeof(bogus); i < len; ++i) {
if (!isspace(line[i])) {
blank = false;
break;
}
}
if (blank) {
continue;
}
/* We got a line with *something* on it. So process it */
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
argv = opal_argv_split(line, ' ');
argc = opal_argv_count(argv);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (argc > 0) {
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
/* Create a temporary env to use in the recursive call --
that is: don't disturb the original env so that we can
have a consistent global env. This allows for the
case:
- Once again: uninteresting cleanup to get diff smaller. This commit was SVN r7178.
2005-09-05 00:54:19 +04:00
orterun --mca foo bar --appfile file
Fix a problem where orterun itself would not receive MCA parameters that were set on the command line. This was techinically exactly the way the code was designed, but it certainly violated the Law of Least Astonishment (even to its designer ;-) ). So now if you execute something like this: mpirun -mca pls_rsh_debug 1 -np 4 hello You'll see debugging output from the rsh pls component, as you would expect (this was not previously the case -- the MCA pls_rsh_debug parame would be set to 1 in the 4 spawned hello processes, but *not* in the orterun process). More specifically, MCA parameters will be set in the orterun process in the following cases: - The new command line switch "--gmca" (or "-gmca") is used, indicating that the MCA parameter is "global". --gmca also means that that MCA parameter will be applied to all context app's. For example: mpirun -gmca foo bar -np 1 hello : -np 2 goodbye The foo MCA param will be set in both the hello and goodbye processes. - If there is only one context app. For example: mpirun -mca pls_rsh_debug 1 -np 4 hello will set pls_rsh_debug to 1 in both the orterun process and the 4 spawned hello processes. Also added a few more comments inside orterun to document a somewhat confusing use of a state variable in a recursive case. This commit was SVN r6764.
2005-08-08 20:42:28 +04:00
where the "file" contains multiple apps. In this case,
each app in "file" will get *only* foo=bar as the base
environment from which its specific environment is
constructed. */
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
if (NULL != *env) {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
tmp_env = opal_argv_copy(*env);
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
if (NULL == tmp_env) {
return ORTE_ERR_OUT_OF_RESOURCE;
}
} else {
tmp_env = NULL;
}
rc = create_app(argc, argv, &app, &made_app, &tmp_env);
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (ORTE_SUCCESS != rc) {
/* Assume that the error message has already been
printed; no need to cleanup -- we can just exit */
exit(1);
}
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
if (NULL != tmp_env) {
* rename ompi_argv to opal_argv * rename ompi_basename to opal_basename * rename ompi bitop functions to opal * rename ompi_cmd_line to opal_cmd_line * rename ompi_sizet2int to opal_sizet2int * rename orte_daemon_init to opal_daemon_init * rename ompi_few to opal_few This commit was SVN r6330.
2005-07-04 04:13:44 +04:00
opal_argv_free(tmp_env);
While waiting for fortran compiles... Fixes for orterun in handling different MCA params for different processes (reviewed by Brian): - By design, if you run the following: mpirun --mca foo aaa --mca foo bbb a.out a.out will get a single MCA param for foo with value "aaa,bbb". - However, if you specify multiple apps with different values for the same MCA param, you should expect to get the different values for each app. For example: mpirun --mca foo aaa a.out : --mca foo bbb b.out Should yield a.out with a "foo" param with value "aaa" and b.out with a "foo" param with a value "bbb". - This did not work -- both a.out and b.out would get a "foo" with "aaa,bbb". - This commit fixes this behavior -- now a.out will get aaa and b.out will get bbb. - Additionally, if you mix --mca and and app file, you can have "global" params and per-line-in-the-appfile params. For example: mpirun --mca foo zzzz --app appfile where "appfile" contains: -np 1 --mca bar aaa a.out -np 1 --mca bar bbb b.out In this case, a.out will get foo=zzzz and bar=aaa, and b.out will get foo=zzzz and bar=bbb. Spiffy. Ok, fortran build is done... back to Fortran... sigh... This commit was SVN r5710.
2005-05-13 18:36:36 +04:00
}
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
if (made_app) {
Clean up a minor memory leak This commit was SVN r9413.
2006-03-24 18:28:42 +03:00
app->idx = app_num;
++app_num;
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_add(jdata->apps, app);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
++jdata->num_apps;
* results from initial merge of the tim branch into the trunk. Compiles and ompi_info works, but that's all that has been tested. This commit was SVN r4827.
2005-03-14 23:57:21 +03:00
}
}
} while (!feof(fp));
fclose(fp);
/* All done */
free(filename);
return ORTE_SUCCESS;
}
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
/*
* Process one line from the orte_base_user_debugger MCA param and
* look for that debugger in the path. If we find it, fill in
* new_argv.
*/
static int process(char *orig_line, char *basename, opal_cmd_line_t *cmd_line,
int argc, char **argv, char ***new_argv, int num_procs)
{
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
int ret = ORTE_SUCCESS;
int i, j, count;
char *line = NULL, *tmp = NULL, *full_line = strdup(orig_line);
char **orterun_argv = NULL, **executable_argv = NULL, **line_argv = NULL;
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
char cwd[OPAL_PATH_MAX];
bool used_num_procs = false;
bool single_app = false;
bool fail_needed_executable = false;
line = full_line;
if (NULL == line) {
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
ret = ORTE_ERR_OUT_OF_RESOURCE;
goto out;
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
}
/* Trim off whitespace at the beginning and ending of line */
for (i = 0; '\0' != line[i] && isspace(line[i]); ++line) {
continue;
}
for (i = strlen(line) - 2; i > 0 && isspace(line[i]); ++i) {
line[i] = '\0';
}
if (strlen(line) <= 0) {
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
ret = ORTE_ERROR;
goto out;
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
}
/* Get the tail of the command line (i.e., the user executable /
argv) */
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
opal_cmd_line_get_tail(cmd_line, &i, &executable_argv);
/* Make a new copy of the orterun command line args, without the
orterun token itself, and without the --debug, --debugger, and
-tv flags. */
orterun_argv = opal_argv_copy(argv);
count = opal_argv_count(orterun_argv);
opal_argv_delete(&count, &orterun_argv, 0, 1);
for (i = 0; NULL != orterun_argv[i]; ++i) {
count = opal_argv_count(orterun_argv);
if (0 == strcmp(orterun_argv[i], "-debug") ||
0 == strcmp(orterun_argv[i], "--debug")) {
opal_argv_delete(&count, &orterun_argv, i, 1);
} else if (0 == strcmp(orterun_argv[i], "-tv") ||
0 == strcmp(orterun_argv[i], "--tv")) {
opal_argv_delete(&count, &orterun_argv, i, 1);
} else if (0 == strcmp(orterun_argv[i], "--debugger") ||
0 == strcmp(orterun_argv[i], "-debugger")) {
opal_argv_delete(&count, &orterun_argv, i, 2);
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
}
}
/* Replace @@ tokens - line should never realistically be bigger
than MAX_INT, so just cast to int to remove compiler warning */
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
*new_argv = NULL;
line_argv = opal_argv_split(line, ' ');
if (NULL == line_argv) {
ret = ORTE_ERR_NOT_FOUND;
goto out;
}
for (i = 0; NULL != line_argv[i]; ++i) {
if (0 == strcmp(line_argv[i], "@mpirun@") ||
0 == strcmp(line_argv[i], "@orterun@")) {
opal_argv_append_nosize(new_argv, argv[0]);
} else if (0 == strcmp(line_argv[i], "@mpirun_args@") ||
0 == strcmp(line_argv[i], "@orterun_args@")) {
for (j = 0; NULL != orterun_argv && NULL != orterun_argv[j]; ++j) {
opal_argv_append_nosize(new_argv, orterun_argv[j]);
}
} else if (0 == strcmp(line_argv[i], "@np@")) {
asprintf(&tmp, "%d", num_procs);
opal_argv_append_nosize(new_argv, tmp);
free(tmp);
} else if (0 == strcmp(line_argv[i], "@single_app@")) {
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
/* This token is only a flag; it is not replaced with any
alternate text */
single_app = true;
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
} else if (0 == strcmp(line_argv[i], "@executable@")) {
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
/* If we found the executable, paste it in. Otherwise,
this is a possible error. */
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
if (NULL != executable_argv) {
opal_argv_append_nosize(new_argv, executable_argv[0]);
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
} else {
fail_needed_executable = true;
}
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
} else if (0 == strcmp(line_argv[i], "@executable_argv@")) {
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
/* If we found the tail, paste in the argv. Otherwise,
this is a possible error. */
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
if (NULL != executable_argv) {
for (j = 1; NULL != executable_argv[j]; ++j) {
opal_argv_append_nosize(new_argv, executable_argv[j]);
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
}
} else {
fail_needed_executable = true;
}
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
} else {
/* It wasn't a special token, so just copy it over */
opal_argv_append_nosize(new_argv, line_argv[i]);
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
}
}
/* Can we find argv[0] in the path? */
getcwd(cwd, OPAL_PATH_MAX);
tmp = opal_path_findv((*new_argv)[0], X_OK, environ, cwd);
if (NULL != tmp) {
free(tmp);
/* Ok, we found a good debugger. Check for some error
conditions. */
tmp = opal_argv_join(argv, ' ');
/* We do not support launching a debugger that requires the
-np value if the user did not specify -np on the command
line. */
if (used_num_procs && 0 == num_procs) {
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
free(tmp);
tmp = opal_argv_join(orterun_argv, ' ');
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
orte_show_help("help-orterun.txt", "debugger requires -np",
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
true, (*new_argv)[0], argv[0], tmp,
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
(*new_argv)[0]);
/* Fall through to free / fail, below */
}
/* Some debuggers do not support launching MPMD */
else if (single_app && NULL != strchr(tmp, ':')) {
orte_show_help("help-orterun.txt",
"debugger only accepts single app", true,
(*new_argv)[0], (*new_argv)[0]);
/* Fall through to free / fail, below */
}
/* Some debuggers do not use orterun/mpirun, and therefore
must have an executable to run (e.g., cannot use mpirun's
app context file feature). */
else if (fail_needed_executable) {
orte_show_help("help-orterun.txt",
"debugger requires executable", true,
(*new_argv)[0], argv[0], (*new_argv)[0], argv[0],
(*new_argv)[0]);
/* Fall through to free / fail, below */
}
/* Otherwise, we succeeded. Return happiness. */
else {
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
goto out;
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
}
free(tmp);
}
/* All done -- didn't find it */
opal_argv_free(*new_argv);
*new_argv = NULL;
As noted on the mailing list by Gabriele Fatigati (http://www.open-mpi.org/community/lists/users/2011/01/15427.php), the --tv (and friends) switches to mpirun would effectively munge the orterun command line together and then split it apart again before exec'ing the underlying debugger. We would therefore lose multi-token argv[x] value and split them into multiple tokens. For example: mpirun --tv -np 2 a.out "foo bar" would get launched with "foo" and "bar" as separate arguments; not one argument. This was due to the underlying code joining the argv into a single string and then re-splitting it. This commit removed the argv join; it now does the parsing and re-jigering of the argv by only looking at each individual argv item; multi-word tokens like "foo bar" will never be split into separate tokens. This commit was SVN r24322.
2011-01-28 16:01:06 +03:00
ret = ORTE_ERR_NOT_FOUND;
out:
if (NULL != orterun_argv) {
opal_argv_free(orterun_argv);
}
if (NULL != executable_argv) {
opal_argv_free(executable_argv);
}
if (NULL != line_argv) {
opal_argv_free(line_argv);
}
if (NULL != tmp) {
free(tmp);
}
if (NULL != full_line) {
free(full_line);
}
return ret;
As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component. Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations. This commit was SVN r23360.
2010-07-07 03:35:42 +04:00
}
/**
* Run a user-level debugger
*/
static void run_debugger(char *basename, opal_cmd_line_t *cmd_line,
int argc, char *argv[], int num_procs)
{
int i, id;
char **new_argv = NULL;
char *value, **lines, *env_name;
/* Get the orte_base_debug MCA parameter and search for a debugger
that can run */
id = mca_base_param_find("orte", NULL, "base_user_debugger");
if (id < 0) {
orte_show_help("help-orterun.txt", "debugger-mca-param-not-found",
true);
exit(1);
}
value = NULL;
mca_base_param_lookup_string(id, &value);
if (NULL == value) {
orte_show_help("help-orterun.txt", "debugger-orte_base_user_debugger-empty",
true);
exit(1);
}
/* Look through all the values in the MCA param */
lines = opal_argv_split(value, ':');
free(value);
for (i = 0; NULL != lines[i]; ++i) {
if (ORTE_SUCCESS == process(lines[i], basename, cmd_line, argc, argv,
&new_argv, num_procs)) {
break;
}
}
/* If we didn't find one, abort */
if (NULL == lines[i]) {
orte_show_help("help-orterun.txt", "debugger-not-found", true);
exit(1);
}
opal_argv_free(lines);
/* We found one */
/* cleanup the MPIR arrays in case the debugger doesn't set them */
memset((char*)MPIR_executable_path, 0, MPIR_MAX_PATH_LENGTH);
memset((char*)MPIR_server_arguments, 0, MPIR_MAX_ARG_LENGTH);
/* Set an MCA param so that everyone knows that they are being
launched under a debugger; not all debuggers are consistent
about setting MPIR_being_debugged in both the launcher and the
MPI processes */
env_name = mca_base_param_environ_variable("orte",
"in_parallel_debugger", NULL);
if (NULL != env_name) {
opal_setenv(env_name, "1", true, &environ);
free(env_name);
}
/* Launch the debugger */
execvp(new_argv[0], new_argv);
value = opal_argv_join(new_argv, ' ');
orte_show_help("help-orterun.txt", "debugger-exec-failed",
true, basename, value, new_argv[0]);
free(value);
opal_argv_free(new_argv);
exit(1);
}
Копировать постоянную ссылку