ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
ee6ee7a10f
openmpi/orte/runtime/orte_globals.c

1080 строки
35 KiB
C
Исходник Обычный вид История

Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/*
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* University Research and Technology
* Corporation. All rights reserved.
Add a resilience to ORTE. Allows the runtime to continue after a process (or ORTED) failure. Note that more work will be necessary to allow the MPI layer to take advantage of this. Per RFC: http://www.open-mpi.org/community/lists/devel/2011/06/9299.php This commit was SVN r24815.
2011-06-24 00:38:02 +04:00
* Copyright (c) 2004-2011 The University of Tennessee and The University
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* of Tennessee Research Foundation. All rights
* reserved.
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
* University of Stuttgart. All rights reserved.
* Copyright (c) 2004-2005 The Regents of the University of California.
* All rights reserved.
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
* Copyright (c) 2007-2011 Cisco Systems, Inc. All rights reserved.
Added new solaris sysinfo module. Also added code to assign orte_local_chip_type and orte_local_chip_model in MPI processes it the appropriate sysinfo module found the values on the machine. This commit was SVN r23581.
2010-08-09 23:28:56 +04:00
* Copyright (c) 2009-2010 Oracle and/or its affiliates. All rights reserved.
Add initial support for dynamic allocations. At this time, only Slurm supports the new capability, which will be included in an upcoming release. Add hooks for supporting dynamic allocation and deallocation to support application-driven requests and fault recovery operations. This commit was SVN r27879.
2013-01-20 04:33:42 +04:00
* Copyright (c) 2011-2013 Los Alamos National Security, LLC.
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
* All rights reserved.
Okay, since a certain other RM out there made a fuss about being able to lock their daemons to specified cores, offer the same option here. The MCA param orte_daemon_cores can be used to specify which core(s) you want the orte daemons to use. This will have no bearing on the application procs - unbound will remain unbound, and binding directives will be applied to the apps. Yippee skippee... This commit was SVN r30513.
2014-01-31 03:50:14 +04:00
* Copyright (c) 2013-2014 Intel, Inc. All rights reserved
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "orte_config.h"
#include "orte/constants.h"
#include "orte/types.h"
#ifdef HAVE_SYS_TIME_H
#include <sys/time.h>
#endif
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
#include "opal/mca/dstore/dstore.h"
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#include "opal/mca/hwloc/hwloc.h"
Add missing header file for the full rte case caused by changes in r19471. This commit was SVN r19474. The following SVN revision numbers were found above: r19471 --> open-mpi/ompi@38eb3019195c88797087e99f9456427a020d0402
2008-09-01 21:49:31 +04:00
#include "opal/util/argv.h"
- On the way to get the BTLs split out and lessen dependency on orte: Often, orte/util/show_help.h is included, although no functionality is required -- instead, most often opal_output.h, or orte/mca/rml/rml_types.h Please see orte_show_help_replacement.sh commited next. - Local compilation (Linux/x86_64) w/ -Wimplicit-function-declaration actually showed two *missing* #include "orte/util/show_help.h" in orte/mca/odls/base/odls_base_default_fns.c and in orte/tools/orte-top/orte-top.c Manually added these. Let's have MTT the last word. This commit was SVN r20557.
2009-02-14 05:26:12 +03:00
#include "opal/util/output.h"
Per a good suggestion from Jeff, make the coprocessor mapping more scalable by using a hash table to cache the coprocessor list, and then do a single pass thru the nodes at the end to assign hostid's. Refs trac:3847 This commit was SVN r29439. The following Trac tickets were found above: Ticket 3847 --> https://svn.open-mpi.org/trac/ompi/ticket/3847
2013-10-15 02:01:48 +04:00
#include "opal/class/opal_hash_table.h"
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
#include "opal/class/opal_pointer_array.h"
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
#include "opal/class/opal_value_array.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "opal/dss/dss.h"
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
#include "opal/threads/threads.h"
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/mca/errmgr/errmgr.h"
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
#include "orte/mca/rml/rml.h"
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
#include "orte/util/proc_info.h"
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
#include "orte/util/name_fns.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/runtime.h"
Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm. Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed. This commit was SVN r18664.
2008-06-18 07:15:56 +04:00
#include "orte/runtime/runtime_internals.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/orte_globals.h"
/* need the data type support functions here */
#include "orte/runtime/data_type_support/orte_dt_support.h"
Get things to build and run when --disable-orte is specified This commit was SVN r26263.
2012-04-11 01:50:01 +04:00
/* State Machine */
opal_list_t orte_job_states;
opal_list_t orte_proc_states;
/* a clean output channel without prefix */
int orte_clean_output = -1;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* globals used by RTE */
bool orte_timing;
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
FILE *orte_timing_output = NULL;
Continue to refine the timing instrumentation to identify where launch time is being spent This commit was SVN r20244.
2009-01-12 22:12:58 +03:00
bool orte_timing_details;
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orte_debug_daemons_file_flag = false;
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
bool orte_leave_session_attached;
Implement the seq rmaps module that sequentially maps process ranks to a list hosts in a hostfile. Restore the "do-not-launch" functionality so users can test a mapping without launching it. Add a "do-not-resolve" cmd line flag to mpirun so the opal/util/if.c code does not attempt to resolve network addresses, thus enabling a user to test a hostfile mapping without hanging on network resolve requests. Add a function to hostfile to generate an ordered list of host names from a hostfile This commit was SVN r18190.
2008-04-17 17:50:59 +04:00
bool orte_do_not_launch = false;
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orted_spin_flag = false;
Added new solaris sysinfo module. Also added code to assign orte_local_chip_type and orte_local_chip_model in MPI processes it the appropriate sysinfo module found the values on the machine. This commit was SVN r23581.
2010-08-09 23:28:56 +04:00
char *orte_local_cpu_type = NULL;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
char *orte_local_cpu_model = NULL;
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
char *orte_basename = NULL;
**************************************************************** This change contains a non-mandatory modification of the MPI-RTE interface. Anyone wishing to support coprocessors such as the Xeon Phi may wish to add the required definition and underlying support **************************************************************** Add locality support for coprocessors such as the Intel Xeon Phi. Detecting that we are on a coprocessor inside of a host node isn't straightforward. There are no good "hooks" provided for programmatically detecting that "we are on a coprocessor running its own OS", and the ORTE daemon just thinks it is on another node. However, in order to properly use the Phi's public interface for MPI transport, it is necessary that the daemon detect that it is colocated with procs on the host. So we have to split the locality to separately record "on the same host" vs "on the same board". We already have the board-level locality flag, but not quite enough flexibility to handle this use-case. Thus, do the following: 1. add OPAL_PROC_ON_HOST flag to indicate we share a host, but not necessarily the same board 2. modify OPAL_PROC_ON_NODE to indicate we share both a host AND the same board. Note that we have to modify the OPAL_PROC_ON_LOCAL_NODE macro to explicitly check both conditions 3. add support in opal/mca/hwloc/base/hwloc_base_util.c for the host to check for coprocessors, and for daemons to check to see if they are on a coprocessor. The former is done via hwloc, but support for the latter is not yet provided by hwloc. So the code for detecting we are on a coprocessor currently is Xeon Phi specific - hopefully, we will find more generic methods in the future. 4. modify the orted and the hnp startup so they check for coprocessors and to see if they are on a coprocessor, and have the orteds pass that info back in their callback message. Automatically detect that coprocessors have been found and identify which coprocessors are on which hosts. Note that this algo isn't scalable at the moment - this will hopefully be improved over time. 5. modify the ompi proc locality detection function to look for coprocessor host info IF the OMPI_RTE_HOST_ID database key has been defined. RTE's that choose not to provide this support do not have to do anything - the associated code will simply be ignored. 6. include some cleanup of the hwloc open/close code so it conforms to how we did things in other frameworks (e.g., having a single "frame" file instead of open/close). Also, fix the locality flags - e.g., being on the same node means you must also be on the same cluster/cu, so ensure those flags are also set. cmr:v1.7.4:reviewer=hjelmn This commit was SVN r29435.
2013-10-14 20:52:58 +04:00
bool orte_coprocessors_detected = false;
Initialize orte_coprocessors hash table to NULL. Delay coprocessor detection on HNP until after node topology final definition in case rmaps changes it. Minor spacing change. Refs trac:3847 This commit was SVN r29504. The following Trac tickets were found above: Ticket 3847 --> https://svn.open-mpi.org/trac/ompi/ticket/3847
2013-10-24 04:08:47 +04:00
opal_hash_table_t *orte_coprocessors = NULL;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
/* ORTE OOB port flags */
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orte_static_ports = false;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
char *orte_oob_static_ports = NULL;
When directly launched by an RM, flag that a process is operating without daemons - i.e., standalone. Provide an error string for the new socket_not_available error. Use errmgr.abort to exit when we cannot get a socket, and ensure that the slurmd module returns the proper exit status for slurm 2.0 This commit was SVN r21868.
2009-08-22 06:58:20 +04:00
bool orte_standalone_operation = false;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
Cleanup some ignores Add missing variables! This commit was SVN r18063.
2008-04-02 00:32:17 +04:00
bool orte_keep_fqdn_hostnames = false;
A better way of handling fqdn allocations. Prior method was wrong as it equated "node1" with "node10", which definitely caused problems. Detect the addition of fqdn nodes in the allocation. If not found, then strip all incoming hostnames from daemons of any domain info when matching those names against the names in the node pool. Leave some protection and "live" diagnostic output in place so we can continue to detect problems across all environments. This commit was SVN r25557.
2011-12-01 18:24:43 +04:00
bool orte_have_fqdn_allocation = false;
Per request from IBM/Eclipse, provide MCA param to request output when nodes are resolved to a different nodename. This really only happens for the node that mpirun executes on, but they need the alert so they can do string matching of node names. This commit was SVN r20032.
2008-11-24 22:57:08 +03:00
bool orte_show_resolved_nodenames;
Add an MCA param to retain all aliases based on IP addrs for node names so that procs can look them up by interface, if desired. If the param is set, pass aliases around to all daemons and procs for local use This commit was SVN r27619.
2012-11-16 08:04:29 +04:00
bool orte_retain_aliases;
In order to properly connect to debuggers like STAT, we need to get the hostname in its unstripped version for the MPIR_proctab. Unfortunately, we need a stripped version for Cray's alps launcher. So when we are stripping the hostname prefix, retain alias hostnames and add the ability to specify an alias to use in the proctab. This commit was SVN r27863.
2013-01-18 09:00:05 +04:00
int orte_use_hostname_alias;
Add an MCA param to retain all aliases based on IP addrs for node names so that procs can look them up by interface, if desired. If the param is set, pass aliases around to all daemons and procs for local use This commit was SVN r27619.
2012-11-16 08:04:29 +04:00
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
int orted_debug_failure;
int orted_debug_failure_delay;
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
bool orte_homogeneous_nodes = false;
bool orte_hetero_apps = false;
Minimize the amount of topology info returned by the daemons. Most clusters, especially at scale, use the same node topology on every node, so there is no re ason to return the topology from every daemon. Borrow a page from the --hetero-apps page and let users indicate that the node topology differs by adding a -- hetero-nodes option to mpirun. If the option is set, then every daemon returns topology info. If not set, then only daemon vpid=1 returns it. We always want one daemon to return the topology as the head node is often different from the compute nodes. Having one daemon return the compute node topolo gy allows us to detect any such difference. All compute nodes are then set to the same topology. This commit was SVN r25408.
2011-11-01 22:43:10 +04:00
bool orte_hetero_nodes = false;
When we hit an error prior to actually launching daemons, it would be nice if orterun didn't bark about daemons failing to launch, mpirun detecting a job failed, etc. Add a new job state to indicate that we never attempted to launch. Flag such a scenario and avoid hitting all the other error messages. This commit was SVN r19366.
2008-08-19 19:19:30 +04:00
bool orte_never_launched = false;
Modify the --display-map and --display-alloc per note on devel list to reduce info for user understanding. Add --display-devel-map and --display-devel-alloc to display all the detailed info we used to provide - it is only of use/interest to developers anyway and confuses users. This commit was SVN r19608.
2008-09-23 19:46:34 +04:00
bool orte_devel_level_output = false;
Add a simple pattern mapper as an example of how to use the topology info to create desired mappings. Let the user specify a pattern based on resource types, and map that pattern across all available nodes as resources permit. Don't automatically display the topology for each node when --display-devel-map is set as it can overwhelm the reader. Use a separate flag --display-topo to get it. This commit was SVN r25396.
2011-10-29 19:12:45 +04:00
bool orte_display_topo_with_map = false;
Add an option for printing a diffable process map for testing mappers This commit was SVN r25428.
2011-11-03 18:22:07 +04:00
bool orte_display_diffable_output = false;
Fully implement the inbound binomial allgather for daemon-based collectives. Supports both modex and barrier operations. Comm_spawn still uses the rank=0 method - shifting that algo to the daemons is under study. This commit was SVN r18115.
2008-04-10 02:10:53 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
char **orte_launch_environ;
Ensure that we only launch procs on the HNP if that node is actually included in the allocation. This commit was SVN r19038.
2008-07-25 21:13:22 +04:00
bool orte_hnp_is_allocated = false;
Establish an MCA param "orte_allocation_required" so that a system can require the user have an RM-provided allocation in order to run. This helps prevent the problem where a user forgets to get an allocation on an RM-managed cluster, and then executes mpirun on the head node - thus causing all of their mpi procs to launch on the head node, usually bringing it to its knees. Since OMPI allows mpirun to default to the local node, and since users want to retain the option to co-locate procs with mpirun, we needed another param to block this error case. This commit was SVN r19135.
2008-08-04 18:25:19 +04:00
bool orte_allocation_required;
Complete the revamp of hostfile support in non-managed environments. Working at the app level, ensure that we utilize only those nodes specified for that app, but fall back to the default hostfile (if available) for those with no specification, further falling back to the local host if the default hostfile is not present or is empty. This commit was SVN r27230.
2012-09-04 20:34:05 +04:00
bool orte_managed_allocation = false;
If (and only if) a user requests, set the default number of slots on any node to the number of objects of the specified type. This *only* takes effect in an unmanaged environment - i.e., if an external resource manager assigns us a number of slots, then that is what we use. However, if we are using a hostfile, then the user may or may not have given us a value for the number of slots on each node. For those nodes (and *only* those nodes) where the user does *not* specify a slot count, we will set the number of slots according to their direction: either to the number of cores, numas, sockets, or hwthreads. Otherwise, the slot count is set to 1. Note that the default behavior remains unchanged: in the absence of any value for #slots, and in the absence of any directive to set #slots, we will set #slots=1. This commit was SVN r27236.
2012-09-05 00:58:26 +04:00
char *orte_set_slots = NULL;
bool orte_display_allocation;
bool orte_display_devel_allocation;
Add an option to treat dash-host specifications as "requested, but not required". So-called "soft" location requests can allow an application to execute even if the ideal allocation isn't available. This commit was SVN r27242.
2012-09-05 22:42:09 +04:00
bool orte_soft_locations = false;
Ensure that we only launch procs on the HNP if that node is actually included in the allocation. This commit was SVN r19038.
2008-07-25 21:13:22 +04:00
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
/* launch agents */
A bunch of minor ORTE valgrind-inspired memory leak cleanups (reviewed by Ralph). This commit was SVN r20544.
2009-02-13 07:14:10 +03:00
char *orte_launch_agent = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
char **orted_cmd_line=NULL;
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
char **orte_fork_agent=NULL;
Modify the wakeup system to make it more Windows-friendly. This allows Shiqing to consolidate the Windows-specific modifications into one location, and generalizes the wakeup procedure in case we hit other system-specific requirements. This needs some soak time to ensure we haven't opened any race conditions. I tried to loop everything in the shutdown procedure through that trigger event call to ensure it all goes through the one-time locks as it did before so that someone hitting ctrl-c when we are already shutting down shouldn't cause problems. Just want to let people use it for awhile to verify. This commit was SVN r19159.
2008-08-05 19:09:29 +04:00
Fix debugger attach This commit was SVN r23921.
2010-10-23 00:07:24 +04:00
/* debugger job */
My apologies for doing this outside of the usual time restrictions, but we need to get this in so we can make progress. Move the ORTE-level debugger code back into orterun and out of the ORTE library to resolve symbol conflicts. This commit was SVN r25713.
2012-01-11 19:53:09 +04:00
bool orte_debugger_dump_proctable;
char *orte_debugger_test_daemon;
bool orte_debugger_test_attach;
int orte_debugger_check_rate;
Enable co-location of debugger daemons during initial launch and when debugging a running job. Provide support for four MPIR extensions that allow specification of debugger daemon executable, argv for the debugger daemon, whether or not to forward debugger daemon IO, and whether or not debugger daemon will piggy-back on ORTE OOB network. Last is not yet implemented. No change in behavior or operation occurs unless (a) the debugger specifically utilizes the extensions and, for co-locate while running, the user specifically enables the capability via an MCA param. Two of the MPIR extensions supported here are used in a widely-used debugger for a large-scale installation. The other two extensions are new and being utilized in prototype work by several debuggers for possible future release. This commit was SVN r19275.
2008-08-13 21:47:24 +04:00
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
/* exit flags */
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
int orte_exit_status = 0;
bool orte_abnormal_term_ordered = false;
Routing must be enabled by default This commit was SVN r25657.
2011-12-15 21:13:52 +04:00
bool orte_routing_is_enabled = true;
Separate abnormal job termination from abnormal orted termination so we can continue to use xcast for orted cmds, but can know to turn off reading of stdin as the job is being terminated. This commit was SVN r20650.
2009-02-27 13:16:25 +03:00
bool orte_job_term_ordered = false;
Ignore comm failures when shutting down orteds This commit was SVN r23201.
2010-05-23 06:57:03 +04:00
bool orte_orteds_term_ordered = false;
Add a new "test" component to the DFS that treats all files as remote in order to test the app-to-daemon interactions on a single machine. Set a global param to indicate we are using staged execution. Add a param to indicate it is okay for non-MPI processes to execute without finalizing. Cleanup file map load and fetch operations. This commit was SVN r27587.
2012-11-10 18:09:12 +04:00
bool orte_allowed_exit_without_sync = false;
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
int orte_startup_timeout;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
int orte_timeout_usec_per_proc;
float orte_max_timeout;
Per request from Dave Goodell, add support for MPIEXEC_TIMEOUT - if set in the environment, terminate the job after the specified number of seconds has passed. Equivalent to MPICH functionality. cmr=v1.7.4:reviewer=dgoodell:subject=add support for MPIEXEC_TIMEOUT This commit was SVN r29831.
2013-12-07 05:58:32 +04:00
orte_timer_t *orte_mpiexec_timeout = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Fix tree-spawn to work within the new modex system This commit was SVN r18349.
2008-05-01 23:19:34 +04:00
opal_buffer_t *orte_tree_launch_cmd = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* global arrays for data storage */
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_t *orte_job_data;
opal_pointer_array_t *orte_node_pool;
Per the RFC from Jeff, move hwloc from opal/mca/common to its own static framework ala libevent. Have ORTE daemons collect the topology info at startup and, if --enable-hwloc-xml is set, send that info back to the HNP for later use. The HNP only retains unique topology "templates" to reduce memory footprint. Have the daemon include the local topology info in the nidmap buffer sent to each app so the apps don't all hammer the local system to discover it for themselves. Remove the sysinfo framework as hwloc replaces that functionality. This commit was SVN r25124.
2011-09-11 23:02:24 +04:00
opal_pointer_array_t *orte_node_topologies;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
opal_pointer_array_t *orte_local_children;
Per request from Nathan, add an offset value to the job struct so we can construct a "global rank" that spans multiple jobs during dynamic launch operations. Store a new ORTE_DB_GLOBAL_RANK value for each process in the database, and ensure that we share our own value during connect_accept so both sides can see it. This isn't being used yet - just enabling Nathan to do what he needs. ***** NOTE: any use of the OMPI_DB_GLOBAL_RANK database key must be protected by #ifdef OMPI_DB_GLOBAL_RANK as not all RTE's will define this key. ***** This commit was SVN r29708.
2013-11-14 21:01:43 +04:00
orte_vpid_t orte_total_procs = 0;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
/* IOF controls */
bool orte_tag_output;
bool orte_timestamp_output;
char *orte_output_filename;
/* generate new xterm windows to display output from specified ranks */
char *orte_xterm;
Add the ability to forward SIGTSTP (converted to SIGSTOP) and SIGCONT to the a.outs. By default, they are not forwarded and the behavior remains as it has always been. However, if one runs with --mca orte_forward_job_control 1, then mpirun will catch those two signals and forward them to the orteds which will deliver them to the a.outs. We have had requests for this feature. This commit was SVN r20391.
2009-01-30 21:50:10 +03:00
/* whether or not to forward SIGTSTP and SIGCONT signals */
bool orte_forward_job_control;
Provide a "progress meter" for launch that outputs progress as we are launching, especially on large jobs. Also, provide a timeout mechanism so that we cleanly abort if we don't get a response from the next daemon in a specified time. This commit was SVN r21359.
2009-06-03 03:52:02 +04:00
/* report launch progress */
bool orte_report_launch_progress = false;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
/* allocation specification */
Correct a couple of bugs in the rank_file mapper that were incorrectly assigning vpids. Add a capability to parse the rankfile to extract node information in place of requiring both hostfile and rankfile for non-RM managed environments. The rankfile is -only- parsed for this IF the hostfile and -host options are not given. Otherwise, those are used to establish allocation info as we did before this commit. This commit was SVN r21815.
2009-08-13 20:08:43 +04:00
char *orte_default_hostfile = NULL;
Allow there to be no default hostfile without generating an error This commit was SVN r25930.
2012-02-15 08:16:05 +04:00
bool orte_default_hostfile_given = false;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
char *orte_rankfile = NULL;
Fix some major bit-rot on scalable launch. If static ports are provided, then daemons can connect back to the HNP via the routed connection tree instead of doing so directly. In order to do that at scale, the node list must be passed as a regular expression - otherwise, the orted command line gets too long. Over the course of time, usage of static ports got corrupted in several places, the "parent" info got incorrectly reset, etc. So correct all that and get the regex-based wireup going again. Also, don't pass node lists if static ports aren't enabled - they are of no value to the orted and just create the possibility of overly-long cmd lines. This commit was SVN r24860.
2011-07-07 22:54:30 +04:00
int orte_num_allocated_nodes = 0;
char *orte_node_regex = NULL;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
Enable a new command line option to --report-events that instructs mpirun to RML-report specific events during job life to the requestor. This commit was SVN r21954.
2009-09-09 09:28:45 +04:00
/* tool communication controls */
bool orte_report_events = false;
char *orte_report_events_uri = NULL;
Instead of an odls-base mca param, make report_bindings a global param so that we can (a) detect it was set in the plm, and then (b) ensure it gets passed along to remote orteds so they will comply with the request. This commit was SVN r22021.
2009-09-28 07:17:15 +04:00
/* report bindings */
bool orte_report_bindings = false;
The ability to add procs to a running job was unfortunately borked when we added the detection of a proc exiting before calling init. Re-enable it here, ensuring that procs that are being restarted and/or added to a job do -not- call barrier during orte_init. This commit was SVN r22404.
2010-01-14 20:59:42 +03:00
/* barrier control */
bool orte_do_not_barrier = false;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
/* process recovery */
bool orte_enable_recovery;
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
int32_t orte_max_restarts;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
/* exit status reporting */
bool orte_report_child_jobs_separately;
struct timeval orte_child_time_to_exit;
Expand our handling of non-zero exit status. If a process exits with non-zero status, pass that info along to the user in case it means something to them, even if the process also exited without calling MPI_Finalize. If the process calls MPI_Abort, that trumps the exit status question. Provide a new MCA param that allows the user to direct that we abort the job once a process exits with non-zero status. No recovery is allowed in such cases to avoid trying to restart a process that has already exited MPI. This commit was SVN r24614.
2011-04-14 19:04:21 +04:00
bool orte_abort_non_zero_exit;
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
/* length of stat history to keep */
int orte_stat_history_size;
Remove stale orte_vm_launch param. Add a param that allows users to specify envars to forward/set so they can do it in the MCA param file instead of only via mpirun cmd line. This commit was SVN r25580.
2011-12-07 01:31:22 +04:00
/* envars to forward */
char *orte_forward_envars = NULL;
Ensure that comm_spawn'd procs get user-specified forwarded envars Thanks to Tim Miller for reporting the regression from the 1.6 series cmr=v1.7.4:reviewer=jsquyres:subject=Ensure that comm_spawn'd procs get user-specified forwarded envars This commit was SVN r30012.
2013-12-20 18:47:35 +04:00
char **orte_forwarded_envars = NULL;
Remove stale orte_vm_launch param. Add a param that allows users to specify envars to forward/set so they can do it in the MCA param file instead of only via mpirun cmd line. This commit was SVN r25580.
2011-12-07 01:31:22 +04:00
Extend the iof by adding two new components to support map-reduce IO chaining. Add a mapreduce tool for running such applications. Fix the state machine to support multiple jobs being simultaneously launched as this is not only required for mapreduce, but can happen under comm-spawn applications as well. This commit was SVN r26380.
2012-05-03 01:00:22 +04:00
/* map-reduce mode */
bool orte_map_reduce = false;
Add a new "test" component to the DFS that treats all files as remote in order to test the app-to-daemon interactions on a single machine. Set a global param to indicate we are using staged execution. Add a param to indicate it is okay for non-MPI processes to execute without finalizing. Cleanup file map load and fetch operations. This commit was SVN r27587.
2012-11-10 18:09:12 +04:00
bool orte_staged_execution = false;
Extend the iof by adding two new components to support map-reduce IO chaining. Add a mapreduce tool for running such applications. Fix the state machine to support multiple jobs being simultaneously launched as this is not only required for mapreduce, but can happen under comm-spawn applications as well. This commit was SVN r26380.
2012-05-03 01:00:22 +04:00
Per request from Brian, enable mapping of stddiag output (output from opal_output calls) to stderr of the local process. This allows you to obtain that output in a local window (for example, when using xterm for each process) instead of having it automatically forwarded to mpirun. Turn this on automatically whenever someone uses the -xterm option, and to be set manually using the orte_map_stddiag_to_stderr mca param. This commit was SVN r26352.
2012-04-27 18:39:34 +04:00
/* map stddiag output to stderr so it isn't forwarded to mpirun */
bool orte_map_stddiag_to_stderr = false;
Allow partial use of allocations by specifying the max number of daemons (i.e., max VM size) for the job This commit was SVN r26499.
2012-05-27 20:48:19 +04:00
/* maximum size of virtual machine - used to subdivide allocation */
int orte_max_vm_size = -1;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
/* progress thread */
opal_thread_t orte_progress_thread;
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
/* global nidmap/pidmap for daemons to give to apps */
opal_byte_object_t orte_nidmap;
opal_byte_object_t orte_pidmap;
MCA/base: Add new MCA variable system Features: - Support for an override parameter file (openmpi-mca-param-override.conf). Variable values in this file can not be overridden by any file or environment value. - Support for boolean, unsigned, and unsigned long long variables. - Support for true/false values. - Support for enumerations on integer variables. - Support for MPIT scope, verbosity, and binding. - Support for command line source. - Support for setting variable source via the environment using OMPI_MCA_SOURCE_<var name>=source (either command or file:filename) - Cleaner API. - Support for variable groups (equivalent to MPIT categories). Notes: - Variables must be created with a backing store (char **, int *, or bool *) that must live at least as long as the variable. - Creating a variable with the MCA_BASE_VAR_FLAG_SETTABLE enables the use of mca_base_var_set_value() to change the value. - String values are duplicated when the variable is registered. It is up to the caller to free the original value if necessary. The new value will be freed by the mca_base_var system and must not be freed by the user. - Variables with constant scope may not be settable. - Variable groups (and all associated variables) are deregistered when the component is closed or the component repository item is freed. This prevents a segmentation fault from accessing a variable after its component is unloaded. - After some discussion we decided we should remove the automatic registration of component priority variables. Few component actually made use of this feature. - The enumerator interface was updated to be general enough to handle future uses of the interface. - The code to generate ompi_info output has been moved into the MCA variable system. See mca_base_var_dump(). opal: update core and components to mca_base_var system orte: update core and components to mca_base_var system ompi: update core and components to mca_base_var system This commit also modifies the rmaps framework. The following variables were moved from ppr and lama: rmaps_base_pernode, rmaps_base_n_pernode, rmaps_base_n_persocket. Both lama and ppr create synonyms for these variables. This commit was SVN r28236.
2013-03-28 01:09:41 +04:00
/* user debugger */
char *orte_base_user_debugger = NULL;
Missed a symbol that needs to exist in non-full RTE case This commit was SVN r19482.
2008-09-02 19:07:48 +04:00
int orte_debug_output = -1;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
bool orte_debug_daemons_flag = false;
bool orte_xml_output = false;
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
FILE *orte_xml_fp = NULL;
Use orte_notifier to alert sys admins to checksum violations in the csum pml. Add ability to store the RM's jobid string to tag the notifier message so that the sys admin knows what job had the problem. This commit was SVN r21687.
2009-07-15 23:43:26 +04:00
char *orte_job_ident = NULL;
Truly respect the -quiet option. Make it an mca param so someone doesn't have to put it solely on the cmd line. Tell show_help to shaddup as well. This commit was SVN r22926.
2010-04-02 18:19:38 +04:00
bool orte_execute_quiet = false;
To help with debugging, add a new mca param that instructs ORTE_ERROR_LOG to output "silent" errors. Helps to track down silent errors that don't have an associated error message (e.g., via show_help). This commit was SVN r23893.
2010-10-16 07:29:47 +04:00
bool orte_report_silent_errors = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Fixes trac:1361: mainly add new internal MCA parameter that orterun will set when it launches under debuggers using the --debug option. This commit was SVN r19116. The following Trac tickets were found above: Ticket 1361 --> https://svn.open-mpi.org/trac/ompi/ticket/1361
2008-08-01 02:11:46 +04:00
/* See comment in orte/tools/orterun/debuggers.c about this MCA
param */
bool orte_in_parallel_debugger = false;
Okay, since a certain other RM out there made a fuss about being able to lock their daemons to specified cores, offer the same option here. The MCA param orte_daemon_cores can be used to specify which core(s) you want the orte daemons to use. This will have no bearing on the application procs - unbound will remain unbound, and binding directives will be applied to the apps. Yippee skippee... This commit was SVN r30513.
2014-01-31 03:50:14 +04:00
char *orte_daemon_cores = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
int orte_dt_init(void)
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
{
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
int rc;
opal_data_type_t tmp;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* set default output */
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
orte_debug_output = opal_output_open(NULL);
Cleanup the output for nodename resolve reporting This commit was SVN r20081.
2008-12-08 22:00:36 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* open up the verbose output for ORTE debugging */
if (orte_debug_flag || 0 < orte_debug_verbosity ||
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
(orte_debug_daemons_flag && (ORTE_PROC_IS_DAEMON || ORTE_PROC_IS_HNP))) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
if (0 < orte_debug_verbosity) {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output_set_verbosity(orte_debug_output, orte_debug_verbosity);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
} else {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output_set_verbosity(orte_debug_output, 1);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
}
}
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/** register the base system types with the DSS */
tmp = ORTE_STD_CNTR;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_std_cntr,
orte_dt_unpack_std_cntr,
(opal_dss_copy_fn_t)orte_dt_copy_std_cntr,
(opal_dss_compare_fn_t)orte_dt_compare_std_cntr,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_STD_CNTR", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NAME;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_name,
orte_dt_unpack_name,
(opal_dss_copy_fn_t)orte_dt_copy_name,
(opal_dss_compare_fn_t)orte_dt_compare_name,
(opal_dss_print_fn_t)orte_dt_print_name,
OPAL_DSS_UNSTRUCTURED,
"ORTE_NAME", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_VPID;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_vpid,
orte_dt_unpack_vpid,
(opal_dss_copy_fn_t)orte_dt_copy_vpid,
(opal_dss_compare_fn_t)orte_dt_compare_vpid,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_VPID", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOBID;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_jobid,
orte_dt_unpack_jobid,
(opal_dss_copy_fn_t)orte_dt_copy_jobid,
(opal_dss_compare_fn_t)orte_dt_compare_jobid,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_JOBID", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
tmp = ORTE_JOB;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_job,
orte_dt_unpack_job,
(opal_dss_copy_fn_t)orte_dt_copy_job,
(opal_dss_compare_fn_t)orte_dt_compare_job,
(opal_dss_print_fn_t)orte_dt_print_job,
OPAL_DSS_STRUCTURED,
"ORTE_JOB", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_node,
orte_dt_unpack_node,
(opal_dss_copy_fn_t)orte_dt_copy_node,
(opal_dss_compare_fn_t)orte_dt_compare_node,
(opal_dss_print_fn_t)orte_dt_print_node,
OPAL_DSS_STRUCTURED,
"ORTE_NODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_PROC;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_proc,
orte_dt_unpack_proc,
(opal_dss_copy_fn_t)orte_dt_copy_proc,
(opal_dss_compare_fn_t)orte_dt_compare_proc,
(opal_dss_print_fn_t)orte_dt_print_proc,
OPAL_DSS_STRUCTURED,
"ORTE_PROC", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_APP_CONTEXT;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_app_context,
orte_dt_unpack_app_context,
(opal_dss_copy_fn_t)orte_dt_copy_app_context,
(opal_dss_compare_fn_t)orte_dt_compare_app_context,
(opal_dss_print_fn_t)orte_dt_print_app_context,
OPAL_DSS_STRUCTURED,
"ORTE_APP_CONTEXT", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NODE_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_node_state,
orte_dt_unpack_node_state,
(opal_dss_copy_fn_t)orte_dt_copy_node_state,
(opal_dss_compare_fn_t)orte_dt_compare_node_state,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_NODE_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_PROC_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_proc_state,
orte_dt_unpack_proc_state,
(opal_dss_copy_fn_t)orte_dt_copy_proc_state,
(opal_dss_compare_fn_t)orte_dt_compare_proc_state,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_PROC_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOB_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_job_state,
orte_dt_unpack_job_state,
(opal_dss_copy_fn_t)orte_dt_copy_job_state,
(opal_dss_compare_fn_t)orte_dt_compare_job_state,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_JOB_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_EXIT_CODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_exit_code,
orte_dt_unpack_exit_code,
(opal_dss_copy_fn_t)orte_dt_copy_exit_code,
(opal_dss_compare_fn_t)orte_dt_compare_exit_code,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_EXIT_CODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOB_MAP;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_map,
orte_dt_unpack_map,
(opal_dss_copy_fn_t)orte_dt_copy_map,
(opal_dss_compare_fn_t)orte_dt_compare_map,
(opal_dss_print_fn_t)orte_dt_print_map,
OPAL_DSS_STRUCTURED,
"ORTE_JOB_MAP", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_RML_TAG;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_tag,
orte_dt_unpack_tag,
(opal_dss_copy_fn_t)orte_dt_copy_tag,
(opal_dss_compare_fn_t)orte_dt_compare_tags,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_RML_TAG", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_DAEMON_CMD;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_daemon_cmd,
orte_dt_unpack_daemon_cmd,
(opal_dss_copy_fn_t)orte_dt_copy_daemon_cmd,
(opal_dss_compare_fn_t)orte_dt_compare_daemon_cmd,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_DAEMON_CMD", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
tmp = ORTE_IOF_TAG;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_iof_tag,
orte_dt_unpack_iof_tag,
(opal_dss_copy_fn_t)orte_dt_copy_iof_tag,
(opal_dss_compare_fn_t)orte_dt_compare_iof_tag,
(opal_dss_print_fn_t)orte_dt_std_print,
OPAL_DSS_UNSTRUCTURED,
"ORTE_IOF_TAG", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Number of small fixes to get the trunk to build again on Catamount This commit was SVN r20141.
2008-12-16 23:09:56 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return ORTE_SUCCESS;
}
orte_job_t* orte_get_job_data_object(orte_jobid_t job)
{
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
int32_t ljob;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Track process heartbeats with time_t, be a little less restrictive on who can retrieve an orte_job_t object This commit was SVN r22921.
2010-03-31 23:20:06 +04:00
/* if the job data wasn't setup, we cannot provide the data */
if (NULL == orte_job_data) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return NULL;
}
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
/* the job is indexed by its local jobid, so we can
* just look it up here. it is not an error for this
* to not be found - could just be
Eliminate an annoying "not found" message when a job abnormally terminates. In this case, we can get a race condition where the job object has been removed, but updates are continuing to flow into the system. This commit was SVN r20857.
2009-03-24 21:06:49 +03:00
* a race condition whereby the job has already been
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
* removed from the array. The get_item function
* will just return NULL in that case.
Eliminate an annoying "not found" message when a job abnormally terminates. In this case, we can get a race condition where the job object has been removed, but updates are continuing to flow into the system. This commit was SVN r20857.
2009-03-24 21:06:49 +03:00
*/
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
ljob = ORTE_LOCAL_JOBID(job);
return (orte_job_t*)opal_pointer_array_get_item(orte_job_data, ljob);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
orte_proc_t* orte_get_proc_object(orte_process_name_t *proc)
{
orte_job_t *jdata;
orte_proc_t *proct;
if (NULL == (jdata = orte_get_job_data_object(proc->jobid))) {
return NULL;
}
proct = (orte_proc_t*)opal_pointer_array_get_item(jdata->procs, proc->vpid);
return proct;
}
orte_vpid_t orte_get_proc_daemon_vpid(orte_process_name_t *proc)
{
orte_job_t *jdata;
orte_proc_t *proct;
if (NULL == (jdata = orte_get_job_data_object(proc->jobid))) {
return ORTE_VPID_INVALID;
}
if (NULL == (proct = (orte_proc_t*)opal_pointer_array_get_item(jdata->procs, proc->vpid))) {
return ORTE_VPID_INVALID;
}
if (NULL == proct->node || NULL == proct->node->daemon) {
return ORTE_VPID_INVALID;
}
return proct->node->daemon->name.vpid;
}
char* orte_get_proc_hostname(orte_process_name_t *proc)
{
orte_proc_t *proct;
char *hostname;
int rc;
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
opal_list_t myvals;
opal_value_t *kv;
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
if (ORTE_PROC_IS_DAEMON || ORTE_PROC_IS_HNP) {
/* look it up on our arrays */
if (NULL == (proct = orte_get_proc_object(proc))) {
ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
return NULL;
}
if (NULL == proct->node || NULL == proct->node->name) {
ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
return NULL;
}
return proct->node->name;
}
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
/* if we are an app, get the data from the modex db */
OBJ_CONSTRUCT(&myvals, opal_list_t);
if (ORTE_SUCCESS != (rc = opal_dstore.fetch(opal_dstore_internal,
(opal_identifier_t*)proc,
ORTE_DB_HOSTNAME,
&myvals))) {
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
ORTE_ERROR_LOG(rc);
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
OPAL_LIST_DESTRUCT(&myvals);
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
return NULL;
}
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
kv = (opal_value_t*)opal_list_get_first(&myvals);
hostname = kv->data.string;
/* protect the data */
kv->data.string = NULL;
OPAL_LIST_DESTRUCT(&myvals);
/* user is responsible for releasing the data */
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
return hostname;
}
orte_node_rank_t orte_get_proc_node_rank(orte_process_name_t *proc)
{
orte_proc_t *proct;
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
orte_node_rank_t noderank;
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
int rc;
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
opal_list_t myvals;
opal_value_t *kv;
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
if (ORTE_PROC_IS_DAEMON || ORTE_PROC_IS_HNP) {
/* look it up on our arrays */
if (NULL == (proct = orte_get_proc_object(proc))) {
ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
return ORTE_NODE_RANK_INVALID;
}
return proct->node_rank;
}
/* if we are an app, get the value from the modex db */
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
OBJ_CONSTRUCT(&myvals, opal_list_t);
if (ORTE_SUCCESS != (rc = opal_dstore.fetch(opal_dstore_internal,
(opal_identifier_t*)proc,
ORTE_DB_NODERANK,
&myvals))) {
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
ORTE_ERROR_LOG(rc);
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
OPAL_LIST_DESTRUCT(&myvals);
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
return ORTE_NODE_RANK_INVALID;
}
As per the RFC: http://www.open-mpi.org/community/lists/devel/2014/04/14496.php Revamp the opal database framework, including renaming it to "dstore" to reflect that it isn't a "database". Move the "db" framework to ORTE for now, soon to move to ORCM This commit was SVN r31557.
2014-04-30 01:49:23 +04:00
kv = (opal_value_t*)opal_list_get_first(&myvals);
noderank = kv->data.uint16;
OPAL_LIST_DESTRUCT(&myvals);
Roll in the rest of the modex change. Eliminate all non-modex API access of RTE info from the MPI layer - in some cases, the info was already present (either in the ompi_proc_t or in the orte_process_info struct) and no call was necessary. This removes all calls to orte_ess from the MPI layer. Calls to orte_grpcomm remain required. Update all the orte ess components to remove their associated APIs for retrieving proc data. Update the grpcomm API to reflect transfer of set/get modex info to the db framework. Note that this doesn't recreate the old GPR. This is strictly a local db storage that may (at some point) obtain any missing data from the local daemon as part of an async methodology. The framework allows us to experiment with such methods without perturbing the default one. This commit was SVN r26678.
2012-06-27 18:53:55 +04:00
return noderank;
}
Add a minor utility This commit was SVN r24379.
2011-02-14 22:45:59 +03:00
orte_vpid_t orte_get_lowest_vpid_alive(orte_jobid_t job)
{
int i;
orte_job_t *jdata;
orte_proc_t *proc;
if (NULL == (jdata = orte_get_job_data_object(job))) {
return ORTE_VPID_INVALID;
}
if (ORTE_PROC_IS_DAEMON &&
ORTE_PROC_MY_NAME->jobid == job &&
NULL != orte_process_info.my_hnp_uri) {
/* if we were started by an HNP, then the lowest vpid
* is always 1
*/
return 1;
}
for (i=0; i < jdata->procs->size; i++) {
if (NULL == (proc = (orte_proc_t*)opal_pointer_array_get_item(jdata->procs, i))) {
continue;
}
if (proc->state == ORTE_PROC_STATE_RUNNING) {
/* must be lowest one alive */
return proc->name.vpid;
}
}
/* only get here if no live proc found */
return ORTE_VPID_INVALID;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
/*
* CONSTRUCTORS, DESTRUCTORS, AND CLASS INSTANTIATIONS
* FOR ORTE CLASSES
*/
static void orte_app_context_construct(orte_app_context_t* app_context)
{
app_context->idx=0;
app_context->app=NULL;
app_context->num_procs=0;
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
OBJ_CONSTRUCT(&app_context->procs, opal_pointer_array_t);
opal_pointer_array_init(&app_context->procs,
1,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
16);
app_context->state = ORTE_APP_STATE_UNDEF;
Implement the MPI_Info part of MPI-3 Ticket 313. Add an MPI_info object MPI_INFO_GET_ENV that contains a number of run-time related pieces of info. This includes all the required ones in the ticket, plus a few that specifically address recent user questions: "num_app_ctx" - the number of app_contexts in the job "first_rank" - the MPI rank of the first process in each app_context "np" - the number of procs in each app_context Still need clarification on the MPI_Init portion of the ticket. Specifically, does the ticket call for returning an error is someone calls MPI_Init more than once in a program? We set a flag to tell us that we have been initialized, but currently never check it. This commit was SVN r27005.
2012-08-12 05:28:23 +04:00
app_context->first_rank = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
app_context->argv=NULL;
app_context->env=NULL;
app_context->cwd=NULL;
app_context->user_specified_cwd=false;
Per discussion with Josh, use the --preload-xxx cmd line options to broadcast files to all nodes. Add --set-cwd-to-session-dir option to start procs in their session directories. Add OMPI_FILE_LOCATION envar to tell procs where their prepositioned files went. This commit was SVN r27125.
2012-08-24 01:28:05 +04:00
app_context->set_cwd_to_session_dir = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
app_context->hostfile=NULL;
app_context->add_hostfile=NULL;
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
app_context->add_host = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
app_context->dash_host = NULL;
Enable the ability to constrain applications to hosts on the basis of resources. This commit was SVN r24736.
2011-05-29 02:18:19 +04:00
OBJ_CONSTRUCT(&app_context->resource_constraints, opal_list_t);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
app_context->prefix_dir = NULL;
app_context->preload_binary = false;
app_context->preload_files = NULL;
Improve the efficiency of the launch system by changing the outer loop to being over app_context, and adding a flag to the app_context so the daemon can record that "this app is on my node" when decoding the launch msg. If the --wdir option is given, check to see if the user provided a relative path. If so, convert it to an absolute path. This is needed to maintain consistent behavior across environements. Some environments automatically chdir to your current working directory when launching the remote orted, while others (e.g., ssh) don't. This levels the playing field and reduces user surprise. This commit was SVN r20342.
2009-01-25 15:39:24 +03:00
app_context->used_on_node = false;
We want uid/gid support at the individual application level. Ensure the values get initialized and packed/unpacked for transfer. This commit was SVN r24489.
2011-03-04 21:46:43 +03:00
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_FT_CR == 1
app_context->sstore_load = NULL;
#endif
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
app_context->recovery_defined = false;
app_context->max_restarts = -1000;
Silence warnings by making types match This commit was SVN r27446.
2012-10-14 07:45:28 +04:00
app_context->max_procs_per_node = 0;
Add initial support for dynamic allocations. At this time, only Slurm supports the new capability, which will be included in an upcoming release. Add hooks for supporting dynamic allocation and deallocation to support application-driven requests and fault recovery operations. This commit was SVN r27879.
2013-01-20 04:33:42 +04:00
app_context->mandatory = false;
app_context->min_number_of_nodes = -1; /* no minimum */
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_app_context_destructor(orte_app_context_t* app_context)
{
Enable the ability to constrain applications to hosts on the basis of resources. This commit was SVN r24736.
2011-05-29 02:18:19 +04:00
opal_list_item_t *item;
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
int i;
orte_proc_t *proc;
Enable the ability to constrain applications to hosts on the basis of resources. This commit was SVN r24736.
2011-05-29 02:18:19 +04:00
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != app_context->app) {
free (app_context->app);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->app = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
for (i=0; i < app_context->procs.size; i++) {
if (NULL != (proc = (orte_proc_t*)opal_pointer_array_get_item(&app_context->procs, i))) {
OBJ_RELEASE(proc);
}
}
OBJ_DESTRUCT(&app_context->procs);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
/* argv and env lists created by util/argv copy functions */
if (NULL != app_context->argv) {
opal_argv_free(app_context->argv);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->argv = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->env) {
opal_argv_free(app_context->env);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->env = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->cwd) {
free (app_context->cwd);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->cwd = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->hostfile) {
free(app_context->hostfile);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->hostfile = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->add_hostfile) {
free(app_context->add_hostfile);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->add_hostfile = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
if (NULL != app_context->add_host) {
opal_argv_free(app_context->add_host);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->add_host = NULL;
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != app_context->dash_host) {
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
opal_argv_free(app_context->dash_host);
app_context->dash_host = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Enable the ability to constrain applications to hosts on the basis of resources. This commit was SVN r24736.
2011-05-29 02:18:19 +04:00
while (NULL != (item = opal_list_remove_first(&app_context->resource_constraints))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&app_context->resource_constraints);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != app_context->prefix_dir) {
free(app_context->prefix_dir);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->prefix_dir = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
app_context->preload_binary = false;
Add a new MPI_Info key to preposition OMPI libraries - implementation underway, but this just defines and passes the new key This commit was SVN r21425.
2009-06-12 21:53:13 +04:00
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if(NULL != app_context->preload_files) {
free(app_context->preload_files);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->preload_files = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_FT_CR == 1
if( NULL != app_context->sstore_load ) {
free(app_context->sstore_load);
app_context->sstore_load = NULL;
}
#endif
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_app_context_t,
opal_object_t,
orte_app_context_construct,
orte_app_context_destructor);
static void orte_job_construct(orte_job_t* job)
{
job->jobid = ORTE_JOBID_INVALID;
Per request from Nathan, add an offset value to the job struct so we can construct a "global rank" that spans multiple jobs during dynamic launch operations. Store a new ORTE_DB_GLOBAL_RANK value for each process in the database, and ensure that we share our own value during connect_accept so both sides can see it. This isn't being used yet - just enabling Nathan to do what he needs. ***** NOTE: any use of the OMPI_DB_GLOBAL_RANK database key must be protected by #ifdef OMPI_DB_GLOBAL_RANK as not all RTE's will define this key. ***** This commit was SVN r29708.
2013-11-14 21:01:43 +04:00
job->offset = 0;
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
job->updated = true;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->apps = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(job->apps,
1,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
2);
job->num_apps = 0;
Default job controls to forward IO. Adjust debugger code to not forward IO unless requested. This commit was SVN r19690.
2008-10-06 21:56:23 +04:00
job->controls = ORTE_JOB_CONTROL_FORWARD_OUTPUT;
Revise the launch timeout detection so we don't mistakenly declare "failed to start". Recognize that timeout is at the per-job level, and define the timeout param as a total value instead of seconds/daemon as it otherwise can get to be an enormous (and useless) number. Resolves problems in loop_spawn where the timer was incorrectly firing and killing the overall job. cmr=v1.7.4:reviewer=hjelmn This commit was SVN r29661.
2013-11-12 03:50:40 +04:00
job->failure_timer = NULL;
This confusion has been around for awhile, caused by a long-ago decision to track slots allocated to a specific job as opposed to allocated to the overall mpirun instance. We eliminated that quite a while ago, but never consolidated the "slots_alloc" and "slots" fields in orte_node_t. As a result, confusion has grown in the code base as to which field to look at and/or update. So (finally) consolidate these two fields into one "slots" field. Add a field in orte_job_t to indicate when all the procs for a job will be launched together, so that staged operations can know when MPI operations are allowed. This commit was SVN r27239.
2012-09-05 05:30:39 +04:00
job->gang_launched = true;
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
job->stdin_target = ORTE_VPID_INVALID;
Extend the iof by adding two new components to support map-reduce IO chaining. Add a mapreduce tool for running such applications. Fix the state machine to support multiple jobs being simultaneously launched as this is not only required for mapreduce, but can happen under comm-spawn applications as well. This commit was SVN r26380.
2012-05-03 01:00:22 +04:00
job->stdout_target = ORTE_JOBID_INVALID;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->total_slots_alloc = 0;
job->num_procs = 0;
job->procs = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(job->procs,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
job->map = NULL;
job->bookmark = NULL;
job->state = ORTE_JOB_STATE_UNDEF;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
job->restart = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Get staged execution working on multi-node setups. Improve efficiency by only remapping if all procs not yet mapped in the job. This commit was SVN r27181.
2012-08-30 00:35:52 +04:00
job->num_mapped = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->num_launched = 0;
job->num_reported = 0;
job->num_terminated = 0;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
job->num_daemons_reported = 0;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
job->num_non_zero_exit = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->abort = false;
job->aborted_proc = NULL;
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
job->originator.jobid = ORTE_JOBID_INVALID;
job->originator.vpid = ORTE_VPID_INVALID;
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
job->recovery_defined = false;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
job->enable_recovery = false;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
job->num_local_procs = 0;
Add a new "test" component to the DFS that treats all files as remote in order to test the app-to-daemon interactions on a single machine. Set a global param to indicate we are using staged execution. Add a param to indicate it is okay for non-MPI processes to execute without finalizing. Cleanup file map load and fetch operations. This commit was SVN r27587.
2012-11-10 18:09:12 +04:00
job->file_maps = NULL;
Enable orterun to directly support staged execution, treating each app as a separate job. Support transfer of file maps when support exists. This commit was SVN r27516.
2012-10-30 03:11:30 +04:00
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->ckpt_state = 0;
job->ckpt_snapshot_ref = NULL;
job->ckpt_snapshot_loc = NULL;
#endif
}
static void orte_job_destruct(orte_job_t* job)
{
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
orte_proc_t *proc;
orte_app_context_t *app;
orte_job_t *jdata;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
int n;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
Some cleanups required from last night's commits to resolve some race conditions and ensure we cleanup properly. Also, remove some debug output that was unintentionally left "on" by default. This commit was SVN r21202.
2009-05-11 18:03:07 +04:00
if (NULL == job) {
/* probably just a race condition - just return */
return;
}
if (orte_debug_flag) {
opal_output(0, "%s Releasing job data for %s",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), ORTE_JOBID_PRINT(job->jobid));
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
for (n=0; n < job->apps->size; n++) {
if (NULL == (app = (orte_app_context_t*)opal_pointer_array_get_item(job->apps, n))) {
continue;
}
OBJ_RELEASE(app);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(job->apps);
Revise the launch timeout detection so we don't mistakenly declare "failed to start". Recognize that timeout is at the per-job level, and define the timeout param as a total value instead of seconds/daemon as it otherwise can get to be an enormous (and useless) number. Resolves problems in loop_spawn where the timer was incorrectly firing and killing the overall job. cmr=v1.7.4:reviewer=hjelmn This commit was SVN r29661.
2013-11-12 03:50:40 +04:00
if (NULL != job->failure_timer) {
OBJ_RELEASE(job->failure_timer);
}
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != job->map) {
OBJ_RELEASE(job->map);
job->map = NULL;
}
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
for (n=0; n < job->procs->size; n++) {
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
if (NULL == (proc = (orte_proc_t*)opal_pointer_array_get_item(job->procs, n))) {
continue;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
OBJ_RELEASE(proc);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(job->procs);
Add a new "test" component to the DFS that treats all files as remote in order to test the app-to-daemon interactions on a single machine. Set a global param to indicate we are using staged execution. Add a param to indicate it is okay for non-MPI processes to execute without finalizing. Cleanup file map load and fetch operations. This commit was SVN r27587.
2012-11-10 18:09:12 +04:00
if (NULL != job->file_maps) {
OBJ_RELEASE(job->file_maps);
Enable orterun to directly support staged execution, treating each app as a separate job. Support transfer of file maps when support exists. This commit was SVN r27516.
2012-10-30 03:11:30 +04:00
}
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != job->ckpt_snapshot_ref) {
free(job->ckpt_snapshot_ref);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
job->ckpt_snapshot_ref = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != job->ckpt_snapshot_loc) {
free(job->ckpt_snapshot_loc);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
job->ckpt_snapshot_loc = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
#endif
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
/* find the job in the global array */
Some cleanups required from last night's commits to resolve some race conditions and ensure we cleanup properly. Also, remove some debug output that was unintentionally left "on" by default. This commit was SVN r21202.
2009-05-11 18:03:07 +04:00
if (NULL != orte_job_data) {
for (n=0; n < orte_job_data->size; n++) {
if (NULL == (jdata = (orte_job_t*)opal_pointer_array_get_item(orte_job_data, n))) {
continue;
}
if (jdata->jobid == job->jobid) {
/* set the entry to NULL */
opal_pointer_array_set_item(orte_job_data, n, NULL);
break;
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
}
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_job_t,
opal_list_item_t,
orte_job_construct,
orte_job_destruct);
static void orte_node_construct(orte_node_t* node)
{
**************************************************************** This change contains a non-mandatory modification of the MPI-RTE interface. Anyone wishing to support coprocessors such as the Xeon Phi may wish to add the required definition and underlying support **************************************************************** Add locality support for coprocessors such as the Intel Xeon Phi. Detecting that we are on a coprocessor inside of a host node isn't straightforward. There are no good "hooks" provided for programmatically detecting that "we are on a coprocessor running its own OS", and the ORTE daemon just thinks it is on another node. However, in order to properly use the Phi's public interface for MPI transport, it is necessary that the daemon detect that it is colocated with procs on the host. So we have to split the locality to separately record "on the same host" vs "on the same board". We already have the board-level locality flag, but not quite enough flexibility to handle this use-case. Thus, do the following: 1. add OPAL_PROC_ON_HOST flag to indicate we share a host, but not necessarily the same board 2. modify OPAL_PROC_ON_NODE to indicate we share both a host AND the same board. Note that we have to modify the OPAL_PROC_ON_LOCAL_NODE macro to explicitly check both conditions 3. add support in opal/mca/hwloc/base/hwloc_base_util.c for the host to check for coprocessors, and for daemons to check to see if they are on a coprocessor. The former is done via hwloc, but support for the latter is not yet provided by hwloc. So the code for detecting we are on a coprocessor currently is Xeon Phi specific - hopefully, we will find more generic methods in the future. 4. modify the orted and the hnp startup so they check for coprocessors and to see if they are on a coprocessor, and have the orteds pass that info back in their callback message. Automatically detect that coprocessors have been found and identify which coprocessors are on which hosts. Note that this algo isn't scalable at the moment - this will hopefully be improved over time. 5. modify the ompi proc locality detection function to look for coprocessor host info IF the OMPI_RTE_HOST_ID database key has been defined. RTE's that choose not to provide this support do not have to do anything - the associated code will simply be ignored. 6. include some cleanup of the hwloc open/close code so it conforms to how we did things in other frameworks (e.g., having a single "frame" file instead of open/close). Also, fix the locality flags - e.g., being on the same node means you must also be on the same cluster/cu, so ensure those flags are also set. cmr:v1.7.4:reviewer=hjelmn This commit was SVN r29435.
2013-10-14 20:52:58 +04:00
node->index = -1;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->name = NULL;
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
node->alias = NULL;
Per a good suggestion from Jeff, make the coprocessor mapping more scalable by using a hash table to cache the coprocessor list, and then do a single pass thru the nodes at the end to assign hostid's. Refs trac:3847 This commit was SVN r29439. The following Trac tickets were found above: Ticket 3847 --> https://svn.open-mpi.org/trac/ompi/ticket/3847
2013-10-15 02:01:48 +04:00
node->serial_number = NULL;
**************************************************************** This change contains a non-mandatory modification of the MPI-RTE interface. Anyone wishing to support coprocessors such as the Xeon Phi may wish to add the required definition and underlying support **************************************************************** Add locality support for coprocessors such as the Intel Xeon Phi. Detecting that we are on a coprocessor inside of a host node isn't straightforward. There are no good "hooks" provided for programmatically detecting that "we are on a coprocessor running its own OS", and the ORTE daemon just thinks it is on another node. However, in order to properly use the Phi's public interface for MPI transport, it is necessary that the daemon detect that it is colocated with procs on the host. So we have to split the locality to separately record "on the same host" vs "on the same board". We already have the board-level locality flag, but not quite enough flexibility to handle this use-case. Thus, do the following: 1. add OPAL_PROC_ON_HOST flag to indicate we share a host, but not necessarily the same board 2. modify OPAL_PROC_ON_NODE to indicate we share both a host AND the same board. Note that we have to modify the OPAL_PROC_ON_LOCAL_NODE macro to explicitly check both conditions 3. add support in opal/mca/hwloc/base/hwloc_base_util.c for the host to check for coprocessors, and for daemons to check to see if they are on a coprocessor. The former is done via hwloc, but support for the latter is not yet provided by hwloc. So the code for detecting we are on a coprocessor currently is Xeon Phi specific - hopefully, we will find more generic methods in the future. 4. modify the orted and the hnp startup so they check for coprocessors and to see if they are on a coprocessor, and have the orteds pass that info back in their callback message. Automatically detect that coprocessors have been found and identify which coprocessors are on which hosts. Note that this algo isn't scalable at the moment - this will hopefully be improved over time. 5. modify the ompi proc locality detection function to look for coprocessor host info IF the OMPI_RTE_HOST_ID database key has been defined. RTE's that choose not to provide this support do not have to do anything - the associated code will simply be ignored. 6. include some cleanup of the hwloc open/close code so it conforms to how we did things in other frameworks (e.g., having a single "frame" file instead of open/close). Also, fix the locality flags - e.g., being on the same node means you must also be on the same cluster/cu, so ensure those flags are also set. cmr:v1.7.4:reviewer=hjelmn This commit was SVN r29435.
2013-10-14 20:52:58 +04:00
node->hostid = ORTE_VPID_INVALID;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->daemon = NULL;
node->daemon_launched = false;
Extend the iof by adding two new components to support map-reduce IO chaining. Add a mapreduce tool for running such applications. Fix the state machine to support multiple jobs being simultaneously launched as this is not only required for mapreduce, but can happen under comm-spawn applications as well. This commit was SVN r26380.
2012-05-03 01:00:22 +04:00
node->location_verified = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->launch_id = -1;
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
node->mapped = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->num_procs = 0;
node->procs = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(node->procs,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
node->next_node_rank = 0;
node->oversubscribed = false;
node->state = ORTE_NODE_STATE_UNKNOWN;
node->slots = 0;
If (and only if) a user requests, set the default number of slots on any node to the number of objects of the specified type. This *only* takes effect in an unmanaged environment - i.e., if an external resource manager assigns us a number of slots, then that is what we use. However, if we are using a hostfile, then the user may or may not have given us a value for the number of slots on each node. For those nodes (and *only* those nodes) where the user does *not* specify a slot count, we will set the number of slots according to their direction: either to the number of cores, numas, sockets, or hwthreads. Otherwise, the slot count is set to 1. Note that the default behavior remains unchanged: in the absence of any value for #slots, and in the absence of any directive to set #slots, we will set #slots=1. This commit was SVN r27236.
2012-09-05 00:58:26 +04:00
node->slots_given = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->slots_inuse = 0;
node->slots_max = 0;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->username = NULL;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
Per the RFC from Jeff, move hwloc from opal/mca/common to its own static framework ala libevent. Have ORTE daemons collect the topology info at startup and, if --enable-hwloc-xml is set, send that info back to the HNP for later use. The HNP only retains unique topology "templates" to reduce memory footprint. Have the daemon include the local topology info in the nidmap buffer sent to each app so the apps don't all hammer the local system to discover it for themselves. Remove the sysinfo framework as hwloc replaces that functionality. This commit was SVN r25124.
2011-09-11 23:02:24 +04:00
#if OPAL_HAVE_HWLOC
node->topology = NULL;
#endif
Extend/update how we handle process stats. Add the ability to collect node-level stats separate from the process stats. Update the process stat memory fields to report in MBytes instead of KBytes as I can't find any process that runs in KBytes nowadays. Rename the memusage sensor plugin to "resusage" as it will soon be updated to include full process stat monitoring. Extend the heartbeat sensor to report node and process stats in the heartbeat. Store the process and node stats in their respective orte_xxx_t object. This commit was SVN r24629.
2011-04-22 02:55:45 +04:00
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
OBJ_CONSTRUCT(&node->stats, opal_ring_buffer_t);
opal_ring_buffer_init(&node->stats, orte_stat_history_size);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_node_destruct(orte_node_t* node)
{
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
int i;
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
opal_node_stats_t *stats;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
orte_proc_t *proc;
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != node->name) {
free(node->name);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->name = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Per a good suggestion from Jeff, make the coprocessor mapping more scalable by using a hash table to cache the coprocessor list, and then do a single pass thru the nodes at the end to assign hostid's. Refs trac:3847 This commit was SVN r29439. The following Trac tickets were found above: Ticket 3847 --> https://svn.open-mpi.org/trac/ompi/ticket/3847
2013-10-15 02:01:48 +04:00
if (NULL != node->serial_number) {
free(node->serial_number);
node->serial_number = NULL;
}
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
if (NULL != node->alias) {
opal_argv_free(node->alias);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->alias = NULL;
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
}
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
if (NULL != node->daemon) {
node->daemon->node = NULL;
OBJ_RELEASE(node->daemon);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->daemon = NULL;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
for (i=0; i < node->procs->size; i++) {
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
if (NULL != (proc = (orte_proc_t*)opal_pointer_array_get_item(node->procs, i))) {
opal_pointer_array_set_item(node->procs, i, NULL);
OBJ_RELEASE(proc);
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(node->procs);
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
/* we release the topology elsewhere */
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != node->username) {
free(node->username);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->username = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
Per the RFC from Jeff, move hwloc from opal/mca/common to its own static framework ala libevent. Have ORTE daemons collect the topology info at startup and, if --enable-hwloc-xml is set, send that info back to the HNP for later use. The HNP only retains unique topology "templates" to reduce memory footprint. Have the daemon include the local topology info in the nidmap buffer sent to each app so the apps don't all hammer the local system to discover it for themselves. Remove the sysinfo framework as hwloc replaces that functionality. This commit was SVN r25124.
2011-09-11 23:02:24 +04:00
/* do NOT destroy the topology */
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
while (NULL != (stats = (opal_node_stats_t*)opal_ring_buffer_pop(&node->stats))) {
OBJ_RELEASE(stats);
}
OBJ_DESTRUCT(&node->stats);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_node_t,
opal_list_item_t,
orte_node_construct,
orte_node_destruct);
static void orte_proc_construct(orte_proc_t* proc)
{
proc->name = *ORTE_NAME_INVALID;
proc->pid = 0;
Continue implementation of resilient operations by supporting reuse of jobids for restarted procs. Ensure that restarted processes have valid node and local ranks, and that node rank values are passed to direct-launched processes. This commit was SVN r21385.
2009-06-06 05:08:47 +04:00
proc->local_rank = ORTE_LOCAL_RANK_INVALID;
proc->node_rank = ORTE_NODE_RANK_INVALID;
Track the app_context rank - will equal overall rank for single app_context jobs This commit was SVN r24778.
2011-06-17 00:31:30 +04:00
proc->app_rank = -1;
ErrMgr Framework redesign to better support fault tolerance development activities. Explained in more detail in the following RFC: http://www.open-mpi.org/community/lists/devel/2010/03/7589.php This commit was SVN r22872.
2010-03-24 00:28:02 +03:00
proc->last_errmgr_state = ORTE_PROC_STATE_UNDEF;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->state = ORTE_PROC_STATE_UNDEF;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
proc->alive = false;
Some minor cleanups. Get singletons working. Cleanup abort handling so it gets properly identified. This commit was SVN r26261.
2012-04-10 23:08:54 +04:00
proc->aborted = false;
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
proc->updated = true;
Change the app_idx type to uint32_t to support users who use large numbers of app_contexts. Set it up as a new typedef so we can change it later without as much effort. This commit was SVN r22727.
2010-02-27 20:37:34 +03:00
proc->app_idx = 0;
Add a simple pattern mapper as an example of how to use the topology info to create desired mappings. Let the user specify a pattern based on resource types, and map that pattern across all available nodes as resources permit. Don't automatically display the topology for each node when --display-devel-map is set as it can overwhelm the reader. Use a separate flag --display-topo to get it. This commit was SVN r25396.
2011-10-29 19:12:45 +04:00
#if OPAL_HAVE_HWLOC
proc->locale = NULL;
Correctly compute usage for dynamic spawns when binding is invoked. Ensure we correctly account for existing process usage on each node when computing bindings during dynamic spawns. cmr=v1.7.4:reviewer=hjelmn:subject=Correctly compute usage for dynamic spawns when binding is invoked This commit was SVN r29649.
2013-11-10 04:38:01 +04:00
proc->bind_location = NULL;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
proc->cpu_bitmap = NULL;
Add a simple pattern mapper as an example of how to use the topology info to create desired mappings. Let the user specify a pattern based on resource types, and map that pattern across all available nodes as resources permit. Don't automatically display the topology for each node when --display-devel-map is set as it can overwhelm the reader. Use a separate flag --display-topo to get it. This commit was SVN r25396.
2011-10-29 19:12:45 +04:00
#endif
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->node = NULL;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
proc->local_proc = false;
proc->do_not_barrier = false;
Update resilient recovery mapping algorithm to be a bit more sophisticated. Track the prior node a proc was on so we avoid ricochet effect. Also avoid putting recovering proc onto node that is already occupied by a peer as this degrades fault tolerance. This commit was SVN r24417.
2011-02-20 21:46:21 +03:00
proc->prior_node = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->nodename = NULL;
Set the return code from the processes upstream. This commit was SVN r24483.
2011-03-03 03:02:21 +03:00
proc->exit_code = 0; /* Assume we won't fail unless otherwise notified */
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->rml_uri = NULL;
Add the ability to track how many times a process has been restarted, and to communicate that value to a process when it is restarted in case it needs to take action when it is restarted as opposed to being started for the first time. This commit was SVN r22377.
2010-01-07 04:19:44 +03:00
proc->restarts = 0;
Add support for detecting rapid failures so the errmgr can respond accordingly This commit was SVN r24796.
2011-06-21 20:08:41 +04:00
proc->fast_failures = 0;
proc->last_failure.tv_sec = 0;
proc->last_failure.tv_usec = 0;
Update heartbeat system This commit was SVN r24404.
2011-02-16 21:50:51 +03:00
proc->reported = false;
Record the number of heartbeats recvd each period for diag purposes This commit was SVN r24714.
2011-05-20 04:21:33 +04:00
proc->beat = 0;
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
OBJ_CONSTRUCT(&proc->stats, opal_ring_buffer_t);
opal_ring_buffer_init(&proc->stats, orte_stat_history_size);
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
proc->registered = false;
Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine. Remove some stale configure.m4's we no longer need. Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages. This commit was SVN r27165.
2012-08-29 01:20:17 +04:00
proc->mpi_proc = false;
Sorry for mid-day commit, but I had promised on the call to do this upon my return. Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code. Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch. This commit was SVN r26242.
2012-04-06 18:23:13 +04:00
proc->deregistered = false;
proc->iof_complete = false;
proc->waitpid_recvd = false;
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->ckpt_state = 0;
proc->ckpt_snapshot_ref = NULL;
proc->ckpt_snapshot_loc = NULL;
#endif
}
static void orte_proc_destruct(orte_proc_t* proc)
{
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
opal_pstats_t *stats;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
/* do NOT free the nodename field as this is
* simply a pointer to a field in the
* associated node object - the node object
* will free it
*/
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#if OPAL_HAVE_HWLOC
if (NULL != proc->cpu_bitmap) {
free(proc->cpu_bitmap);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
#endif
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != proc->node) {
OBJ_RELEASE(proc->node);
proc->node = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != proc->rml_uri) {
free(proc->rml_uri);
proc->rml_uri = NULL;
}
Add an mca param to set the "fork agent" - i.e., a program to be run when forking off a process (e.g., valgrind). While you could specify this by "mpirun -n N fork_agent ./my_app", not everyone launches procs with ORTE from mpirun. Provide the ability to store recent stat histories using the ring_buffer class This commit was SVN r24842.
2011-06-30 07:12:38 +04:00
while (NULL != (stats = (opal_pstats_t*)opal_ring_buffer_pop(&proc->stats))) {
OBJ_RELEASE(stats);
}
Extend/update how we handle process stats. Add the ability to collect node-level stats separate from the process stats. Update the process stat memory fields to report in MBytes instead of KBytes as I can't find any process that runs in KBytes nowadays. Rename the memusage sensor plugin to "resusage" as it will soon be updated to include full process stat monitoring. Extend the heartbeat sensor to report node and process stats in the heartbeat. Store the process and node stats in their respective orte_xxx_t object. This commit was SVN r24629.
2011-04-22 02:55:45 +04:00
OBJ_DESTRUCT(&proc->stats);
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != proc->ckpt_snapshot_ref) {
free(proc->ckpt_snapshot_ref);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->ckpt_snapshot_ref = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != proc->ckpt_snapshot_loc) {
free(proc->ckpt_snapshot_loc);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->ckpt_snapshot_loc = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
#endif
}
OBJ_CLASS_INSTANCE(orte_proc_t,
opal_list_item_t,
orte_proc_construct,
orte_proc_destruct);
static void orte_job_map_construct(orte_job_map_t* map)
{
George raised some valid concerns about the extensibility of the revised rmaps framework. Address those by: 1. removing the enum of mapper values 2. change the req_mapper and last_mapper fields to char* so they can hold the component name instead of a mapper flag 3. revise the selection logic in the mapper components to reflect the change. Components now look for their name in the req_mapper field, or to see if other criteria (e.g., npernode) are set that mandate their doing the mapping Several MCA params resided in the rmaps base for historical reasons - they have been in the base since at least the original 1.2 release (and perhaps earlier). However, George correctly pointed out that they really should reside in their respective components. Accordingly, move them to the components, but register synonyms to the old names to avoid breaking backward compatibility. These revisions retain the current functionality of allowing comm_spawn'd jobs to use different mappers than the original job, and for the errmgr to utilize the resilient mapper to recover processes regardless of how they were originally mapped. Given the large number of possible combinations, I am sure that someone will find a corner-case combination of values and selection criteria that cause either no mapper to be selected, or one other than the intended to be used. No one can test all the ways people will use this system, so I expect debugging to continue for awhile. The ability of comm_spawn'd jobs to exploit this functionality relies on changes to the orte_dpm component - this will be committed separately. This commit was SVN r24520.
2011-03-12 08:30:09 +03:00
map->req_mapper = NULL;
map->last_mapper = NULL;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
map->mapping = 0;
map->ranking = 0;
#if OPAL_HAVE_HWLOC
map->binding = 0;
#endif
map->ppr = NULL;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
map->cpus_per_rank = 1;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
map->display_map = false;
map->num_new_daemons = 0;
map->daemon_vpid_start = ORTE_VPID_INVALID;
map->num_nodes = 0;
map->nodes = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(map->nodes,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
}
static void orte_job_map_destruct(orte_job_map_t* map)
{
orte_std_cntr_t i;
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
orte_node_t *node;
George raised some valid concerns about the extensibility of the revised rmaps framework. Address those by: 1. removing the enum of mapper values 2. change the req_mapper and last_mapper fields to char* so they can hold the component name instead of a mapper flag 3. revise the selection logic in the mapper components to reflect the change. Components now look for their name in the req_mapper field, or to see if other criteria (e.g., npernode) are set that mandate their doing the mapping Several MCA params resided in the rmaps base for historical reasons - they have been in the base since at least the original 1.2 release (and perhaps earlier). However, George correctly pointed out that they really should reside in their respective components. Accordingly, move them to the components, but register synonyms to the old names to avoid breaking backward compatibility. These revisions retain the current functionality of allowing comm_spawn'd jobs to use different mappers than the original job, and for the errmgr to utilize the resilient mapper to recover processes regardless of how they were originally mapped. Given the large number of possible combinations, I am sure that someone will find a corner-case combination of values and selection criteria that cause either no mapper to be selected, or one other than the intended to be used. No one can test all the ways people will use this system, so I expect debugging to continue for awhile. The ability of comm_spawn'd jobs to exploit this functionality relies on changes to the orte_dpm component - this will be committed separately. This commit was SVN r24520.
2011-03-12 08:30:09 +03:00
if (NULL != map->req_mapper) {
free(map->req_mapper);
}
if (NULL != map->last_mapper) {
free(map->last_mapper);
}
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
if (NULL != map->ppr) {
free(map->ppr);
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
for (i=0; i < map->nodes->size; i++) {
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here: https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation. In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions: 1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior. 2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation. 3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so. As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes. This commit was SVN r25476.
2011-11-15 07:40:11 +04:00
if (NULL != (node = (orte_node_t*)opal_pointer_array_get_item(map->nodes, i))) {
OBJ_RELEASE(node);
opal_pointer_array_set_item(map->nodes, i, NULL);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
}
OBJ_RELEASE(map->nodes);
}
OBJ_CLASS_INSTANCE(orte_job_map_t,
opal_object_t,
orte_job_map_construct,
orte_job_map_destruct);
Копировать постоянную ссылку