ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
257473ebca
openmpi/orte/runtime/orte_globals.c

1128 строки
38 KiB
C
Исходник Обычный вид История

Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/*
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* University Research and Technology
* Corporation. All rights reserved.
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
* Copyright (c) 2004-2008 The University of Tennessee and The University
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* of Tennessee Research Foundation. All rights
* reserved.
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
* University of Stuttgart. All rights reserved.
* Copyright (c) 2004-2005 The Regents of the University of California.
* All rights reserved.
A bunch of minor ORTE valgrind-inspired memory leak cleanups (reviewed by Ralph). This commit was SVN r20544.
2009-02-13 07:14:10 +03:00
* Copyright (c) 2007-2009 Cisco Systems, Inc. All rights reserved.
Added new solaris sysinfo module. Also added code to assign orte_local_chip_type and orte_local_chip_model in MPI processes it the appropriate sysinfo module found the values on the machine. This commit was SVN r23581.
2010-08-09 23:28:56 +04:00
* Copyright (c) 2009-2010 Oracle and/or its affiliates. All rights reserved.
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "orte_config.h"
#include "orte/constants.h"
#include "orte/types.h"
#ifdef HAVE_SYS_TIME_H
#include <sys/time.h>
#endif
#include "opal/mca/base/mca_base_param.h"
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
#include "opal/mca/paffinity/paffinity.h"
Add missing header file for the full rte case caused by changes in r19471. This commit was SVN r19474. The following SVN revision numbers were found above: r19471 --> open-mpi/ompi@38eb3019195c88797087e99f9456427a020d0402
2008-09-01 21:49:31 +04:00
#include "opal/util/argv.h"
- On the way to get the BTLs split out and lessen dependency on orte: Often, orte/util/show_help.h is included, although no functionality is required -- instead, most often opal_output.h, or orte/mca/rml/rml_types.h Please see orte_show_help_replacement.sh commited next. - Local compilation (Linux/x86_64) w/ -Wimplicit-function-declaration actually showed two *missing* #include "orte/util/show_help.h" in orte/mca/odls/base/odls_base_default_fns.c and in orte/tools/orte-top/orte-top.c Manually added these. Let's have MTT the last word. This commit was SVN r20557.
2009-02-14 05:26:12 +03:00
#include "opal/util/output.h"
* Wrap all the direct error-code checks of the form (OMPI_ERR_* == ret) with (OMPI_ERR_* = OPAL_SOS_GET_ERR_CODE(ret)), since the return value could be a SOS-encoded error. The OPAL_SOS_GET_ERR_CODE() takes in a SOS error and returns back the native error code. * Since OPAL_SUCCESS is preserved by SOS, also change all calls of the form (OPAL_ERROR == ret) to (OPAL_SUCCESS != ret). We thus avoid having to decode 'ret' to get the native error code. This commit was SVN r23162.
2010-05-18 03:08:56 +04:00
#include "opal/util/opal_sos.h"
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
#include "opal/class/opal_pointer_array.h"
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
#include "opal/class/opal_value_array.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "opal/dss/dss.h"
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
#include "opal/threads/threads.h"
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/mca/errmgr/errmgr.h"
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
#include "orte/mca/rml/rml.h"
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
#include "orte/util/proc_info.h"
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
#include "orte/util/name_fns.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/runtime.h"
Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm. Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed. This commit was SVN r18664.
2008-06-18 07:15:56 +04:00
#include "orte/runtime/runtime_internals.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/orte_globals.h"
/* need the data type support functions here */
#include "orte/runtime/data_type_support/orte_dt_support.h"
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* globals used by RTE */
bool orte_timing;
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
FILE *orte_timing_output = NULL;
Continue to refine the timing instrumentation to identify where launch time is being spent This commit was SVN r20244.
2009-01-12 22:12:58 +03:00
bool orte_timing_details;
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orte_debug_daemons_file_flag = false;
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
bool orte_leave_session_attached;
Implement the seq rmaps module that sequentially maps process ranks to a list hosts in a hostfile. Restore the "do-not-launch" functionality so users can test a mapping without launching it. Add a "do-not-resolve" cmd line flag to mpirun so the opal/util/if.c code does not attempt to resolve network addresses, thus enabling a user to test a hostfile mapping without hanging on network resolve requests. Add a function to hostfile to generate an ordered list of host names from a hostfile This commit was SVN r18190.
2008-04-17 17:50:59 +04:00
bool orte_do_not_launch = false;
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orted_spin_flag = false;
Update the cm ess module to support orted bootstrap. Continue work towards bootstrap capability. This commit was SVN r21989.
2009-09-22 06:16:40 +04:00
bool orte_daemon_bootstrap = false;
Added new solaris sysinfo module. Also added code to assign orte_local_chip_type and orte_local_chip_model in MPI processes it the appropriate sysinfo module found the values on the machine. This commit was SVN r23581.
2010-08-09 23:28:56 +04:00
char *orte_local_cpu_type = NULL;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
char *orte_local_cpu_model = NULL;
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
char *orte_basename = NULL;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
/* ORTE OOB port flags */
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orte_static_ports = false;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
char *orte_oob_static_ports = NULL;
When directly launched by an RM, flag that a process is operating without daemons - i.e., standalone. Provide an error string for the new socket_not_available error. Use errmgr.abort to exit when we cannot get a socket, and ensure that the slurmd module returns the proper exit status for slurm 2.0 This commit was SVN r21868.
2009-08-22 06:58:20 +04:00
bool orte_standalone_operation = false;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
Cleanup some ignores Add missing variables! This commit was SVN r18063.
2008-04-02 00:32:17 +04:00
bool orte_keep_fqdn_hostnames = false;
Per request from IBM/Eclipse, provide MCA param to request output when nodes are resolved to a different nodename. This really only happens for the node that mpirun executes on, but they need the alert so they can do string matching of node names. This commit was SVN r20032.
2008-11-24 22:57:08 +03:00
bool orte_show_resolved_nodenames;
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
int orted_debug_failure;
int orted_debug_failure_delay;
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
bool orte_homogeneous_nodes = false;
bool orte_hetero_apps = false;
When we hit an error prior to actually launching daemons, it would be nice if orterun didn't bark about daemons failing to launch, mpirun detecting a job failed, etc. Add a new job state to indicate that we never attempted to launch. Flag such a scenario and avoid hitting all the other error messages. This commit was SVN r19366.
2008-08-19 19:19:30 +04:00
bool orte_never_launched = false;
Modify the --display-map and --display-alloc per note on devel list to reduce info for user understanding. Add --display-devel-map and --display-devel-alloc to display all the detailed info we used to provide - it is only of use/interest to developers anyway and confuses users. This commit was SVN r19608.
2008-09-23 19:46:34 +04:00
bool orte_devel_level_output = false;
Fully implement the inbound binomial allgather for daemon-based collectives. Supports both modex and barrier operations. Comm_spawn still uses the rank=0 method - shifting that algo to the daemons is under study. This commit was SVN r18115.
2008-04-10 02:10:53 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
char **orte_launch_environ;
Ensure that we only launch procs on the HNP if that node is actually included in the allocation. This commit was SVN r19038.
2008-07-25 21:13:22 +04:00
bool orte_hnp_is_allocated = false;
Establish an MCA param "orte_allocation_required" so that a system can require the user have an RM-provided allocation in order to run. This helps prevent the problem where a user forgets to get an allocation on an RM-managed cluster, and then executes mpirun on the head node - thus causing all of their mpi procs to launch on the head node, usually bringing it to its knees. Since OMPI allows mpirun to default to the local node, and since users want to retain the option to co-locate procs with mpirun, we needed another param to block this error case. This commit was SVN r19135.
2008-08-04 18:25:19 +04:00
bool orte_allocation_required;
Ensure that we only launch procs on the HNP if that node is actually included in the allocation. This commit was SVN r19038.
2008-07-25 21:13:22 +04:00
A bunch of minor ORTE valgrind-inspired memory leak cleanups (reviewed by Ralph). This commit was SVN r20544.
2009-02-13 07:14:10 +03:00
char *orte_launch_agent = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
char **orted_cmd_line=NULL;
Modify the wakeup system to make it more Windows-friendly. This allows Shiqing to consolidate the Windows-specific modifications into one location, and generalizes the wakeup procedure in case we hit other system-specific requirements. This needs some soak time to ensure we haven't opened any race conditions. I tried to loop everything in the shutdown procedure through that trigger event call to ensure it all goes through the one-time locks as it did before so that someone hitting ctrl-c when we are already shutting down shouldn't cause problems. Just want to let people use it for awhile to verify. This commit was SVN r19159.
2008-08-05 19:09:29 +04:00
Fix debugger attach This commit was SVN r23921.
2010-10-23 00:07:24 +04:00
/* debugger job */
Enable co-location of debugger daemons during initial launch and when debugging a running job. Provide support for four MPIR extensions that allow specification of debugger daemon executable, argv for the debugger daemon, whether or not to forward debugger daemon IO, and whether or not debugger daemon will piggy-back on ORTE OOB network. Last is not yet implemented. No change in behavior or operation occurs unless (a) the debugger specifically utilizes the extensions and, for co-locate while running, the user specifically enables the capability via an MCA param. Two of the MPIR extensions supported here are used in a widely-used debugger for a large-scale installation. The other two extensions are new and being utilized in prototype work by several debuggers for possible future release. This commit was SVN r19275.
2008-08-13 21:47:24 +04:00
orte_job_t *orte_debugger_daemon=NULL;
Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases. The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang. This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is: * pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place * modified the errmgr to directly call the new routines when termination is detected * removed the grpcomm.onesided_barrier and its associated RML tag * add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree * use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial. This commit was SVN r23429.
2010-07-18 01:03:27 +04:00
/* exit flags */
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
int orte_exit_status = 0;
bool orte_abnormal_term_ordered = false;
This is a first step towards supporting fully-routed OOB communications: 1. remove direct routed module (hooray!) 2. add radix tree routed module (binomial remains default) 3. remove duplicate data storage - orteds were storing nidmap and pidmap data in odls, everyone else in ess 4. add ess APIs to update nidmap, add new pidmap - used only by orteds for MPI-2 support 5. modify code to eliminate multiple calls to orte_routed.update_route that recreated info already in ess pidmap. Add ess API to lookup that info instead. Modify routed modules to utilize that capability 6. setup new ability to shutdown orteds without sending back an "ack" message to mpirun - not utilized yet, will require some changes to plm terminate_orteds functions in managed environments (coming soon) Initial tests indicating that fully routing comm via defined routing trees may not actually have a significant cost for operations like IB QP setup. More tests required to confirm. This will require an autogen... This commit was SVN r19866.
2008-11-01 00:10:00 +03:00
bool orte_routing_is_enabled = false;
Separate abnormal job termination from abnormal orted termination so we can continue to use xcast for orted cmds, but can know to turn off reading of stdin as the job is being terminated. This commit was SVN r20650.
2009-02-27 13:16:25 +03:00
bool orte_job_term_ordered = false;
Ignore comm failures when shutting down orteds This commit was SVN r23201.
2010-05-23 06:57:03 +04:00
bool orte_orteds_term_ordered = false;
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
int orte_startup_timeout;
Fully implement the inbound binomial allgather for daemon-based collectives. Supports both modex and barrier operations. Comm_spawn still uses the rank=0 method - shifting that algo to the daemons is under study. This commit was SVN r18115.
2008-04-10 02:10:53 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
int orte_timeout_usec_per_proc;
float orte_max_timeout;
Fix tree-spawn to work within the new modex system This commit was SVN r18349.
2008-05-01 23:19:34 +04:00
opal_buffer_t *orte_tree_launch_cmd = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* global arrays for data storage */
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_t *orte_job_data;
opal_pointer_array_t *orte_node_pool;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Cleanup the output for nodename resolve reporting This commit was SVN r20081.
2008-12-08 22:00:36 +03:00
/* a clean output channel without prefix */
int orte_clean_output = -1;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
/* RHC: FLAG TO SELECT WHETHER OR NOT TO SEND PROFILE FILE IN NIDMAP */
bool orte_send_profile;
/* Nidmap and job maps */
opal_pointer_array_t orte_nidmap;
opal_pointer_array_t orte_jobmap;
Continue development of regular expression support by implementing it for slurm launches. Works for both initial (cmd line and non-cmd line) and comm_spawn launch. Additional work required to fully enable static port support when using cmd line regular expression launch system. This commit was SVN r21502.
2009-06-24 00:25:38 +04:00
bool orte_use_regexp;
char *orted_launch_cmd = NULL;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
/* list of local children on a daemon */
opal_list_t orte_local_children;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
opal_mutex_t orte_local_children_lock;
opal_condition_t orte_local_children_cond;
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
/* list of job data for local children on a daemon */
opal_list_t orte_local_jobdata;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
opal_mutex_t orte_local_jobdata_lock;
opal_condition_t orte_local_jobdata_cond;
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
/* IOF controls */
bool orte_tag_output;
bool orte_timestamp_output;
char *orte_output_filename;
/* generate new xterm windows to display output from specified ranks */
char *orte_xterm;
Add the ability to forward SIGTSTP (converted to SIGSTOP) and SIGCONT to the a.outs. By default, they are not forwarded and the behavior remains as it has always been. However, if one runs with --mca orte_forward_job_control 1, then mpirun will catch those two signals and forward them to the orteds which will deliver them to the a.outs. We have had requests for this feature. This commit was SVN r20391.
2009-01-30 21:50:10 +03:00
/* whether or not to forward SIGTSTP and SIGCONT signals */
bool orte_forward_job_control;
Enable comm_spawn of slave processes, currently only active for the rsh, slurm, and tm environments. Establish support for local rsh environments in the plm/base so that rsh of local slaves can be done by any environment that supports it. Create new orte_rsh_agent param so users can specify rsh agent from outside of rsh plm, and sym link that to the old plm_rsh_agent and pls_rsh_agent options. Modify the orte-bootproxy to pass prefix for the remote slave to support hetero/hybrid scenarios This commit was SVN r20492.
2009-02-09 23:44:44 +03:00
/* rsh support */
A bunch of minor ORTE valgrind-inspired memory leak cleanups (reviewed by Ralph). This commit was SVN r20544.
2009-02-13 07:14:10 +03:00
char *orte_rsh_agent = NULL;
Add the ability to directly launch procs via rsh/ssh. Collect common functions in plm/base. Create a new global param to set assume_same_shell, alias'd back to plm_rsh_assume_same_shell (not deprecated). This commit was SVN r21328.
2009-05-30 05:10:25 +04:00
bool orte_assume_same_shell = true;
Enable comm_spawn of slave processes, currently only active for the rsh, slurm, and tm environments. Establish support for local rsh environments in the plm/base so that rsh of local slaves can be done by any environment that supports it. Create new orte_rsh_agent param so users can specify rsh agent from outside of rsh plm, and sym link that to the old plm_rsh_agent and pls_rsh_agent options. Modify the orte-bootproxy to pass prefix for the remote slave to support hetero/hybrid scenarios This commit was SVN r20492.
2009-02-09 23:44:44 +03:00
Provide a "progress meter" for launch that outputs progress as we are launching, especially on large jobs. Also, provide a timeout mechanism so that we cleanly abort if we don't get a response from the next daemon in a specified time. This commit was SVN r21359.
2009-06-03 03:52:02 +04:00
/* report launch progress */
bool orte_report_launch_progress = false;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
/* cluster hardware info */
uint8_t orte_default_num_boards;
uint8_t orte_default_num_sockets_per_board;
uint8_t orte_default_num_cores_per_socket;
/* allocation specification */
char *orte_default_cpu_set;
Correct a couple of bugs in the rank_file mapper that were incorrectly assigning vpids. Add a capability to parse the rankfile to extract node information in place of requiring both hostfile and rankfile for non-RM managed environments. The rankfile is -only- parsed for this IF the hostfile and -host options are not given. Otherwise, those are used to establish allocation info as we did before this commit. This commit was SVN r21815.
2009-08-13 20:08:43 +04:00
char *orte_default_hostfile = NULL;
char *orte_rankfile;
Use this option only on Windows, so protect it with #ifdef __WINDOWS__. This commit was SVN r22701.
2010-02-24 11:50:03 +03:00
#ifdef __WINDOWS__
Add a global option for inputing the head node name for Windows CCP trough command line. This commit was SVN r22688.
2010-02-23 22:42:51 +03:00
char *orte_ccp_headnode;
Use this option only on Windows, so protect it with #ifdef __WINDOWS__. This commit was SVN r22701.
2010-02-24 11:50:03 +03:00
#endif
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
/* default rank assigment and binding policy */
orte_mapping_policy_t orte_default_mapping_policy = 0;
Enable a new command line option to --report-events that instructs mpirun to RML-report specific events during job life to the requestor. This commit was SVN r21954.
2009-09-09 09:28:45 +04:00
/* tool communication controls */
bool orte_report_events = false;
char *orte_report_events_uri = NULL;
Instead of an odls-base mca param, make report_bindings a global param so that we can (a) detect it was set in the plm, and then (b) ensure it gets passed along to remote orteds so they will comply with the request. This commit was SVN r22021.
2009-09-28 07:17:15 +04:00
/* report bindings */
bool orte_report_bindings = false;
The ability to add procs to a running job was unfortunately borked when we added the detection of a proc exiting before calling init. Re-enable it here, ensuring that procs that are being restarted and/or added to a job do -not- call barrier during orte_init. This commit was SVN r22404.
2010-01-14 20:59:42 +03:00
/* barrier control */
bool orte_do_not_barrier = false;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
/* process recovery */
bool orte_enable_recovery;
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
int32_t orte_max_restarts;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
/* comm fn for updating state */
orte_default_comm_fn_t orte_comm;
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
/* exit status reporting */
bool orte_report_child_jobs_separately;
struct timeval orte_child_time_to_exit;
Expand our handling of non-zero exit status. If a process exits with non-zero status, pass that info along to the user in case it means something to them, even if the process also exited without calling MPI_Finalize. If the process calls MPI_Abort, that trumps the exit status question. Provide a new MCA param that allows the user to direct that we abort the job once a process exits with non-zero status. No recovery is allowed in such cases to avoid trying to restart a process that has already exited MPI. This commit was SVN r24614.
2011-04-14 19:04:21 +04:00
bool orte_abort_non_zero_exit;
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
Forgot that Terry wanted to control the vm launch with an mca param - set one up for that purpose This commit was SVN r24525.
2011-03-13 03:46:42 +03:00
/* VM control */
bool orte_vm_launch = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#endif /* !ORTE_DISABLE_FULL_RTE */
Missed a symbol that needs to exist in non-full RTE case This commit was SVN r19482.
2008-09-02 19:07:48 +04:00
int orte_debug_output = -1;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
bool orte_debug_daemons_flag = false;
bool orte_xml_output = false;
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
FILE *orte_xml_fp = NULL;
Use orte_notifier to alert sys admins to checksum violations in the csum pml. Add ability to store the RM's jobid string to tag the notifier message so that the sys admin knows what job had the problem. This commit was SVN r21687.
2009-07-15 23:43:26 +04:00
char *orte_job_ident = NULL;
Truly respect the -quiet option. Make it an mca param so someone doesn't have to put it solely on the cmd line. Tell show_help to shaddup as well. This commit was SVN r22926.
2010-04-02 18:19:38 +04:00
bool orte_execute_quiet = false;
To help with debugging, add a new mca param that instructs ORTE_ERROR_LOG to output "silent" errors. Helps to track down silent errors that don't have an associated error message (e.g., via show_help). This commit was SVN r23893.
2010-10-16 07:29:47 +04:00
bool orte_report_silent_errors = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Fixes trac:1361: mainly add new internal MCA parameter that orterun will set when it launches under debuggers using the --debug option. This commit was SVN r19116. The following Trac tickets were found above: Ticket 1361 --> https://svn.open-mpi.org/trac/ompi/ticket/1361
2008-08-01 02:11:46 +04:00
/* See comment in orte/tools/orterun/debuggers.c about this MCA
param */
bool orte_in_parallel_debugger = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
int orte_dt_init(void)
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
{
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
int rc;
opal_data_type_t tmp;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* set default output */
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
orte_debug_output = opal_output_open(NULL);
Cleanup the output for nodename resolve reporting This commit was SVN r20081.
2008-12-08 22:00:36 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* open up the verbose output for ORTE debugging */
if (orte_debug_flag || 0 < orte_debug_verbosity ||
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
(orte_debug_daemons_flag && (ORTE_PROC_IS_DAEMON || ORTE_PROC_IS_HNP))) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
if (0 < orte_debug_verbosity) {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output_set_verbosity(orte_debug_output, orte_debug_verbosity);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
} else {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output_set_verbosity(orte_debug_output, 1);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
}
}
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/** register the base system types with the DSS */
tmp = ORTE_STD_CNTR;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_std_cntr,
orte_dt_unpack_std_cntr,
(opal_dss_copy_fn_t)orte_dt_copy_std_cntr,
(opal_dss_compare_fn_t)orte_dt_compare_std_cntr,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_STD_CNTR", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NAME;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_name,
orte_dt_unpack_name,
(opal_dss_copy_fn_t)orte_dt_copy_name,
(opal_dss_compare_fn_t)orte_dt_compare_name,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_print_name,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_NAME", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_VPID;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_vpid,
orte_dt_unpack_vpid,
(opal_dss_copy_fn_t)orte_dt_copy_vpid,
(opal_dss_compare_fn_t)orte_dt_compare_vpid,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_VPID", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOBID;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_jobid,
orte_dt_unpack_jobid,
(opal_dss_copy_fn_t)orte_dt_copy_jobid,
(opal_dss_compare_fn_t)orte_dt_compare_jobid,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_JOBID", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
tmp = ORTE_JOB;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_job,
orte_dt_unpack_job,
(opal_dss_copy_fn_t)orte_dt_copy_job,
(opal_dss_compare_fn_t)orte_dt_compare_job,
(opal_dss_size_fn_t)orte_dt_size_job,
(opal_dss_print_fn_t)orte_dt_print_job,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_JOB", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_node,
orte_dt_unpack_node,
(opal_dss_copy_fn_t)orte_dt_copy_node,
(opal_dss_compare_fn_t)orte_dt_compare_node,
(opal_dss_size_fn_t)orte_dt_size_node,
(opal_dss_print_fn_t)orte_dt_print_node,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_NODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_PROC;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_proc,
orte_dt_unpack_proc,
(opal_dss_copy_fn_t)orte_dt_copy_proc,
(opal_dss_compare_fn_t)orte_dt_compare_proc,
(opal_dss_size_fn_t)orte_dt_size_proc,
(opal_dss_print_fn_t)orte_dt_print_proc,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_PROC", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_APP_CONTEXT;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_app_context,
orte_dt_unpack_app_context,
(opal_dss_copy_fn_t)orte_dt_copy_app_context,
(opal_dss_compare_fn_t)orte_dt_compare_app_context,
(opal_dss_size_fn_t)orte_dt_size_app_context,
(opal_dss_print_fn_t)orte_dt_print_app_context,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_APP_CONTEXT", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NODE_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_node_state,
orte_dt_unpack_node_state,
(opal_dss_copy_fn_t)orte_dt_copy_node_state,
(opal_dss_compare_fn_t)orte_dt_compare_node_state,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_NODE_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_PROC_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_proc_state,
orte_dt_unpack_proc_state,
(opal_dss_copy_fn_t)orte_dt_copy_proc_state,
(opal_dss_compare_fn_t)orte_dt_compare_proc_state,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_PROC_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOB_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_job_state,
orte_dt_unpack_job_state,
(opal_dss_copy_fn_t)orte_dt_copy_job_state,
(opal_dss_compare_fn_t)orte_dt_compare_job_state,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_JOB_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_EXIT_CODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_exit_code,
orte_dt_unpack_exit_code,
(opal_dss_copy_fn_t)orte_dt_copy_exit_code,
(opal_dss_compare_fn_t)orte_dt_compare_exit_code,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_EXIT_CODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOB_MAP;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_map,
orte_dt_unpack_map,
(opal_dss_copy_fn_t)orte_dt_copy_map,
(opal_dss_compare_fn_t)orte_dt_compare_map,
(opal_dss_size_fn_t)orte_dt_size_map,
(opal_dss_print_fn_t)orte_dt_print_map,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_JOB_MAP", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_RML_TAG;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_tag,
orte_dt_unpack_tag,
(opal_dss_copy_fn_t)orte_dt_copy_tag,
(opal_dss_compare_fn_t)orte_dt_compare_tags,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_RML_TAG", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_DAEMON_CMD;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_daemon_cmd,
orte_dt_unpack_daemon_cmd,
(opal_dss_copy_fn_t)orte_dt_copy_daemon_cmd,
(opal_dss_compare_fn_t)orte_dt_compare_daemon_cmd,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_DAEMON_CMD", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_GRPCOMM_MODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_grpcomm_mode,
orte_dt_unpack_grpcomm_mode,
(opal_dss_copy_fn_t)orte_dt_copy_grpcomm_mode,
(opal_dss_compare_fn_t)orte_dt_compare_grpcomm_mode,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_GRPCOMM_MODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
tmp = ORTE_IOF_TAG;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_iof_tag,
orte_dt_unpack_iof_tag,
(opal_dss_copy_fn_t)orte_dt_copy_iof_tag,
(opal_dss_compare_fn_t)orte_dt_compare_iof_tag,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_IOF_TAG", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Number of small fixes to get the trunk to build again on Catamount This commit was SVN r20141.
2008-12-16 23:09:56 +03:00
#endif /* !ORTE_DISABLE_FULL_SUPPORT */
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return ORTE_SUCCESS;
}
Some fixes for the disabled ORTE case * Protect an orte variable used in the orte debugger stuff * Initialize the datatype code in the Catamount code, as we need it for intercommunicators (the proc code needs it to pack the remote name) * Turn on a bunch of the orte datatype code so that ORTE_NAME is available. This commit was SVN r19457.
2008-08-31 22:06:55 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
orte_job_t* orte_get_job_data_object(orte_jobid_t job)
{
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
int32_t ljob;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Track process heartbeats with time_t, be a little less restrictive on who can retrieve an orte_job_t object This commit was SVN r22921.
2010-03-31 23:20:06 +04:00
/* if the job data wasn't setup, we cannot provide the data */
if (NULL == orte_job_data) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return NULL;
}
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
/* the job is indexed by its local jobid, so we can
* just look it up here. it is not an error for this
* to not be found - could just be
Eliminate an annoying "not found" message when a job abnormally terminates. In this case, we can get a race condition where the job object has been removed, but updates are continuing to flow into the system. This commit was SVN r20857.
2009-03-24 21:06:49 +03:00
* a race condition whereby the job has already been
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
* removed from the array. The get_item function
* will just return NULL in that case.
Eliminate an annoying "not found" message when a job abnormally terminates. In this case, we can get a race condition where the job object has been removed, but updates are continuing to flow into the system. This commit was SVN r20857.
2009-03-24 21:06:49 +03:00
*/
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
ljob = ORTE_LOCAL_JOBID(job);
return (orte_job_t*)opal_pointer_array_get_item(orte_job_data, ljob);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Add a minor utility This commit was SVN r24379.
2011-02-14 22:45:59 +03:00
orte_vpid_t orte_get_lowest_vpid_alive(orte_jobid_t job)
{
int i;
orte_job_t *jdata;
orte_proc_t *proc;
if (NULL == (jdata = orte_get_job_data_object(job))) {
return ORTE_VPID_INVALID;
}
if (ORTE_PROC_IS_DAEMON &&
ORTE_PROC_MY_NAME->jobid == job &&
NULL != orte_process_info.my_hnp_uri) {
/* if we were started by an HNP, then the lowest vpid
* is always 1
*/
return 1;
}
for (i=0; i < jdata->procs->size; i++) {
if (NULL == (proc = (orte_proc_t*)opal_pointer_array_get_item(jdata->procs, i))) {
continue;
}
if (proc->state == ORTE_PROC_STATE_RUNNING) {
/* must be lowest one alive */
return proc->name.vpid;
}
}
/* only get here if no live proc found */
return ORTE_VPID_INVALID;
}
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
int orte_global_comm(orte_process_name_t *recipient,
opal_buffer_t *buf, orte_rml_tag_t tag,
orte_default_cbfunc_t cbfunc)
{
int ret;
if (recipient->jobid == ORTE_PROC_MY_NAME->jobid &&
recipient->vpid == ORTE_PROC_MY_NAME->vpid &&
NULL != cbfunc) {
/* if I am the recipient and a direct fn is provided, use a message event */
ORTE_MESSAGE_EVENT(ORTE_PROC_MY_NAME, buf, tag, cbfunc);
ret = ORTE_SUCCESS;
} else {
/* go ahead and send it */
if (0 > (ret = orte_rml.send_buffer(recipient, buf, tag, 0))) {
ORTE_ERROR_LOG(ret);
} else {
ret = ORTE_SUCCESS;
}
}
return ret;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
/*
* CONSTRUCTORS, DESTRUCTORS, AND CLASS INSTANTIATIONS
* FOR ORTE CLASSES
*/
static void orte_app_context_construct(orte_app_context_t* app_context)
{
app_context->idx=0;
app_context->app=NULL;
app_context->num_procs=0;
app_context->argv=NULL;
app_context->env=NULL;
app_context->cwd=NULL;
app_context->user_specified_cwd=false;
app_context->hostfile=NULL;
app_context->add_hostfile=NULL;
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
app_context->add_host = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
app_context->dash_host = NULL;
app_context->prefix_dir = NULL;
app_context->preload_binary = false;
app_context->preload_files = NULL;
app_context->preload_files_dest_dir = NULL;
Add ability for local slave spawns to pre-position supporting files. Update comm_spawn and comm_spawn_multiple man pages to cover new info_keys. This commit was SVN r20527.
2009-02-12 18:56:45 +03:00
app_context->preload_files_src_dir = NULL;
Improve the efficiency of the launch system by changing the outer loop to being over app_context, and adding a flag to the app_context so the daemon can record that "this app is on my node" when decoding the launch msg. If the --wdir option is given, check to see if the user provided a relative path. If so, convert it to an absolute path. This is needed to maintain consistent behavior across environements. Some environments automatically chdir to your current working directory when launching the remote orted, while others (e.g., ssh) don't. This levels the playing field and reduces user surprise. This commit was SVN r20342.
2009-01-25 15:39:24 +03:00
app_context->used_on_node = false;
We want uid/gid support at the individual application level. Ensure the values get initialized and packed/unpacked for transfer. This commit was SVN r24489.
2011-03-04 21:46:43 +03:00
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_FT_CR == 1
app_context->sstore_load = NULL;
#endif
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
app_context->recovery_defined = false;
app_context->max_restarts = -1000;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_app_context_destructor(orte_app_context_t* app_context)
{
if (NULL != app_context->app) {
free (app_context->app);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->app = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
/* argv and env lists created by util/argv copy functions */
if (NULL != app_context->argv) {
opal_argv_free(app_context->argv);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->argv = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->env) {
opal_argv_free(app_context->env);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->env = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->cwd) {
free (app_context->cwd);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->cwd = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->hostfile) {
free(app_context->hostfile);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->hostfile = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->add_hostfile) {
free(app_context->add_hostfile);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->add_hostfile = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
if (NULL != app_context->add_host) {
opal_argv_free(app_context->add_host);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->add_host = NULL;
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != app_context->dash_host) {
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
opal_argv_free(app_context->dash_host);
app_context->dash_host = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->prefix_dir) {
free(app_context->prefix_dir);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->prefix_dir = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
app_context->preload_binary = false;
Add a new MPI_Info key to preposition OMPI libraries - implementation underway, but this just defines and passes the new key This commit was SVN r21425.
2009-06-12 21:53:13 +04:00
app_context->preload_libs = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if(NULL != app_context->preload_files) {
free(app_context->preload_files);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->preload_files = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if(NULL != app_context->preload_files_dest_dir) {
free(app_context->preload_files_dest_dir);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->preload_files_dest_dir = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add ability for local slave spawns to pre-position supporting files. Update comm_spawn and comm_spawn_multiple man pages to cover new info_keys. This commit was SVN r20527.
2009-02-12 18:56:45 +03:00
if(NULL != app_context->preload_files_src_dir) {
free(app_context->preload_files_src_dir);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->preload_files_src_dir = NULL;
Add ability for local slave spawns to pre-position supporting files. Update comm_spawn and comm_spawn_multiple man pages to cover new info_keys. This commit was SVN r20527.
2009-02-12 18:56:45 +03:00
}
A number of C/R enhancements per RFC below: http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
#if OPAL_ENABLE_FT_CR == 1
if( NULL != app_context->sstore_load ) {
free(app_context->sstore_load);
app_context->sstore_load = NULL;
}
#endif
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_app_context_t,
opal_object_t,
orte_app_context_construct,
orte_app_context_destructor);
static void orte_job_construct(orte_job_t* job)
{
Arggg...revert r24645. I knew those fields were there for a reason...sigh. This commit was SVN r24647. The following SVN revision numbers were found above: r24645 --> open-mpi/ompi@e4732110da2d0d082cc0cc8631159fff16e0048d
2011-04-28 19:07:00 +04:00
job->name = NULL;
job->instance = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->jobid = ORTE_JOBID_INVALID;
job->apps = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(job->apps,
1,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
2);
job->num_apps = 0;
Default job controls to forward IO. Adjust debugger code to not forward IO unless requested. This commit was SVN r19690.
2008-10-06 21:56:23 +04:00
job->controls = ORTE_JOB_CONTROL_FORWARD_OUTPUT;
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
job->stdin_target = ORTE_VPID_INVALID;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->total_slots_alloc = 0;
job->num_procs = 0;
job->procs = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(job->procs,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
job->map = NULL;
job->bookmark = NULL;
job->oversubscribe_override = false;
job->state = ORTE_JOB_STATE_UNDEF;
job->num_launched = 0;
job->num_reported = 0;
job->num_terminated = 0;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
job->num_daemons_reported = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->abort = false;
job->aborted_proc = NULL;
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
Fix a situation where we were unlocking a thread that isn't locked for the main launch - it is only used for dynamic spawns. This commit was SVN r23682.
2010-08-28 18:03:17 +04:00
OBJ_CONSTRUCT(&job->dyn_spawn_lock, opal_mutex_t);
OBJ_CONSTRUCT(&job->dyn_spawn_cond, opal_condition_t);
job->dyn_spawn_active = false;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do. Cleanup the heartbeat handling so it is associated with the proc, not a node. Cleanup handling of recovery options so that defaults do not override user values iff they are provided. This commit was SVN r24382.
2011-02-14 23:49:12 +03:00
job->recovery_defined = false;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
job->enable_recovery = false;
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
job->launch_msg_sent.tv_sec = 0;
job->launch_msg_sent.tv_usec = 0;
job->max_launch_msg_recvd.tv_sec = 0;
job->max_launch_msg_recvd.tv_usec = 0;
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->ckpt_state = 0;
job->ckpt_snapshot_ref = NULL;
job->ckpt_snapshot_loc = NULL;
#endif
}
static void orte_job_destruct(orte_job_t* job)
{
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
orte_proc_t *proc;
orte_app_context_t *app;
orte_job_t *jdata;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
int n;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Some cleanups required from last night's commits to resolve some race conditions and ensure we cleanup properly. Also, remove some debug output that was unintentionally left "on" by default. This commit was SVN r21202.
2009-05-11 18:03:07 +04:00
if (NULL == job) {
/* probably just a race condition - just return */
return;
}
if (orte_debug_flag) {
opal_output(0, "%s Releasing job data for %s",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), ORTE_JOBID_PRINT(job->jobid));
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
Arggg...revert r24645. I knew those fields were there for a reason...sigh. This commit was SVN r24647. The following SVN revision numbers were found above: r24645 --> open-mpi/ompi@e4732110da2d0d082cc0cc8631159fff16e0048d
2011-04-28 19:07:00 +04:00
if (NULL != job->name) {
free(job->name);
}
if (NULL != job->instance) {
free(job->instance);
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
for (n=0; n < job->apps->size; n++) {
if (NULL == (app = (orte_app_context_t*)opal_pointer_array_get_item(job->apps, n))) {
continue;
}
OBJ_RELEASE(app);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(job->apps);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != job->map) {
OBJ_RELEASE(job->map);
job->map = NULL;
}
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
for (n=0; n < job->procs->size; n++) {
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
if (NULL == (proc = (orte_proc_t*)opal_pointer_array_get_item(job->procs, n))) {
continue;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
OBJ_RELEASE(proc);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(job->procs);
Fix a situation where we were unlocking a thread that isn't locked for the main launch - it is only used for dynamic spawns. This commit was SVN r23682.
2010-08-28 18:03:17 +04:00
OBJ_DESTRUCT(&job->dyn_spawn_lock);
OBJ_DESTRUCT(&job->dyn_spawn_cond);
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != job->ckpt_snapshot_ref) {
free(job->ckpt_snapshot_ref);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
job->ckpt_snapshot_ref = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != job->ckpt_snapshot_loc) {
free(job->ckpt_snapshot_loc);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
job->ckpt_snapshot_loc = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
#endif
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
/* find the job in the global array */
Some cleanups required from last night's commits to resolve some race conditions and ensure we cleanup properly. Also, remove some debug output that was unintentionally left "on" by default. This commit was SVN r21202.
2009-05-11 18:03:07 +04:00
if (NULL != orte_job_data) {
for (n=0; n < orte_job_data->size; n++) {
if (NULL == (jdata = (orte_job_t*)opal_pointer_array_get_item(orte_job_data, n))) {
continue;
}
if (jdata->jobid == job->jobid) {
/* set the entry to NULL */
opal_pointer_array_set_item(orte_job_data, n, NULL);
break;
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
}
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_job_t,
opal_list_item_t,
orte_job_construct,
orte_job_destruct);
static void orte_node_construct(orte_node_t* node)
{
node->name = NULL;
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
node->alias = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->index = -1;
node->daemon = NULL;
node->daemon_launched = false;
node->launch_id = -1;
node->num_procs = 0;
node->procs = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(node->procs,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
node->next_node_rank = 0;
node->oversubscribed = false;
node->state = ORTE_NODE_STATE_UNKNOWN;
node->slots = 0;
node->slots_inuse = 0;
node->slots_alloc = 0;
node->slots_max = 0;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
node->boards = orte_default_num_boards;
node->sockets_per_board = orte_default_num_sockets_per_board;
node->cores_per_socket = orte_default_num_cores_per_socket;
if (NULL != orte_default_cpu_set) {
node->cpu_set = strdup(orte_default_cpu_set);
} else {
node->cpu_set = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->username = NULL;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
OBJ_CONSTRUCT(&node->resources, opal_list_t);
Extend/update how we handle process stats. Add the ability to collect node-level stats separate from the process stats. Update the process stat memory fields to report in MBytes instead of KBytes as I can't find any process that runs in KBytes nowadays. Rename the memusage sensor plugin to "resusage" as it will soon be updated to include full process stat monitoring. Extend the heartbeat sensor to report node and process stats in the heartbeat. Store the process and node stats in their respective orte_xxx_t object. This commit was SVN r24629.
2011-04-22 02:55:45 +04:00
OBJ_CONSTRUCT(&node->stats, opal_node_stats_t);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_node_destruct(orte_node_t* node)
{
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
int i;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
opal_list_item_t *item;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != node->name) {
free(node->name);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->name = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
if (NULL != node->alias) {
opal_argv_free(node->alias);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->alias = NULL;
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
}
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
if (NULL != node->daemon) {
node->daemon->node = NULL;
OBJ_RELEASE(node->daemon);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->daemon = NULL;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
for (i=0; i < node->procs->size; i++) {
if (NULL != node->procs->addr[i]) {
((orte_proc_t*)(node->procs->addr[i]))->node = NULL;
OBJ_RELEASE(node->procs->addr[i]);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->procs->addr[i] = NULL;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(node->procs);
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
if (NULL != node->cpu_set) {
free(node->cpu_set);
node->cpu_set = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != node->username) {
free(node->username);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->username = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
while (NULL != (item = opal_list_remove_first(&node->resources))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&node->resources);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_node_t,
opal_list_item_t,
orte_node_construct,
orte_node_destruct);
static void orte_proc_construct(orte_proc_t* proc)
{
proc->name = *ORTE_NAME_INVALID;
proc->pid = 0;
Continue implementation of resilient operations by supporting reuse of jobids for restarted procs. Ensure that restarted processes have valid node and local ranks, and that node rank values are passed to direct-launched processes. This commit was SVN r21385.
2009-06-06 05:08:47 +04:00
proc->local_rank = ORTE_LOCAL_RANK_INVALID;
proc->node_rank = ORTE_NODE_RANK_INVALID;
ErrMgr Framework redesign to better support fault tolerance development activities. Explained in more detail in the following RFC: http://www.open-mpi.org/community/lists/devel/2010/03/7589.php This commit was SVN r22872.
2010-03-24 00:28:02 +03:00
proc->last_errmgr_state = ORTE_PROC_STATE_UNDEF;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->state = ORTE_PROC_STATE_UNDEF;
Change the app_idx type to uint32_t to support users who use large numbers of app_contexts. Set it up as a new typedef so we can change it later without as much effort. This commit was SVN r22727.
2010-02-27 20:37:34 +03:00
proc->app_idx = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->slot_list = NULL;
proc->node = NULL;
Update resilient recovery mapping algorithm to be a bit more sophisticated. Track the prior node a proc was on so we avoid ricochet effect. Also avoid putting recovering proc onto node that is already occupied by a peer as this degrades fault tolerance. This commit was SVN r24417.
2011-02-20 21:46:21 +03:00
proc->prior_node = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->nodename = NULL;
Set the return code from the processes upstream. This commit was SVN r24483.
2011-03-03 03:02:21 +03:00
proc->exit_code = 0; /* Assume we won't fail unless otherwise notified */
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->rml_uri = NULL;
Add the ability to track how many times a process has been restarted, and to communicate that value to a process when it is restarted in case it needs to take action when it is restarted as opposed to being started for the first time. This commit was SVN r22377.
2010-01-07 04:19:44 +03:00
proc->restarts = 0;
Update heartbeat system This commit was SVN r24404.
2011-02-16 21:50:51 +03:00
proc->reported = false;
proc->beat = false;
Extend/update how we handle process stats. Add the ability to collect node-level stats separate from the process stats. Update the process stat memory fields to report in MBytes instead of KBytes as I can't find any process that runs in KBytes nowadays. Rename the memusage sensor plugin to "resusage" as it will soon be updated to include full process stat monitoring. Extend the heartbeat sensor to report node and process stats in the heartbeat. Store the process and node stats in their respective orte_xxx_t object. This commit was SVN r24629.
2011-04-22 02:55:45 +04:00
OBJ_CONSTRUCT(&proc->stats, opal_pstats_t);
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->ckpt_state = 0;
proc->ckpt_snapshot_ref = NULL;
proc->ckpt_snapshot_loc = NULL;
#endif
}
static void orte_proc_destruct(orte_proc_t* proc)
{
/* do NOT free the nodename field as this is
* simply a pointer to a field in the
* associated node object - the node object
* will free it
*/
if (NULL != proc->slot_list) {
free(proc->slot_list);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->slot_list = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != proc->node) {
OBJ_RELEASE(proc->node);
proc->node = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != proc->rml_uri) {
free(proc->rml_uri);
proc->rml_uri = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Extend/update how we handle process stats. Add the ability to collect node-level stats separate from the process stats. Update the process stat memory fields to report in MBytes instead of KBytes as I can't find any process that runs in KBytes nowadays. Rename the memusage sensor plugin to "resusage" as it will soon be updated to include full process stat monitoring. Extend the heartbeat sensor to report node and process stats in the heartbeat. Store the process and node stats in their respective orte_xxx_t object. This commit was SVN r24629.
2011-04-22 02:55:45 +04:00
OBJ_DESTRUCT(&proc->stats);
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != proc->ckpt_snapshot_ref) {
free(proc->ckpt_snapshot_ref);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->ckpt_snapshot_ref = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != proc->ckpt_snapshot_loc) {
free(proc->ckpt_snapshot_loc);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->ckpt_snapshot_loc = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
#endif
}
OBJ_CLASS_INSTANCE(orte_proc_t,
opal_list_item_t,
orte_proc_construct,
orte_proc_destruct);
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
static void orte_attr_construct(orte_attr_t *ptr)
{
ptr->name = NULL;
ptr->size = 0;
ptr->bytes = NULL;
}
static void orte_attr_destruct(orte_attr_t *ptr)
{
if (NULL != ptr->name) {
free(ptr->name);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->name = NULL;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
}
if (NULL != ptr->bytes) {
free(ptr->bytes);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->bytes = NULL;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
}
}
OBJ_CLASS_INSTANCE(orte_attr_t,
opal_list_item_t,
orte_attr_construct,
orte_attr_destruct);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
static void orte_nid_construct(orte_nid_t *ptr)
{
ptr->name = NULL;
ptr->daemon = ORTE_VPID_INVALID;
Not totally necessary, but good form - init the oversubscribed field in the orte_nid_t object This commit was SVN r24127.
2010-12-01 15:58:37 +03:00
ptr->oversubscribed = false;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
OBJ_CONSTRUCT(&ptr->attrs, opal_list_t);
Expand the ESS framework's API to include a new function "query_sys_info" that allows the caller to retrieve key-value pairs of info on the local system capabilities (e.g., cpu type/model). Have each daemon and the HNP "sense" that information and provide it to their local procs to avoid having every proc querying the system directly. This commit was SVN r22870.
2010-03-23 23:47:41 +03:00
OBJ_CONSTRUCT(&ptr->sysinfo, opal_list_t);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_nid_destruct(orte_nid_t *ptr)
{
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
opal_list_item_t *item;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != ptr->name) {
free(ptr->name);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->name = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
while (NULL != (item = opal_list_remove_first(&ptr->attrs))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&ptr->attrs);
Expand the ESS framework's API to include a new function "query_sys_info" that allows the caller to retrieve key-value pairs of info on the local system capabilities (e.g., cpu type/model). Have each daemon and the HNP "sense" that information and provide it to their local procs to avoid having every proc querying the system directly. This commit was SVN r22870.
2010-03-23 23:47:41 +03:00
while (NULL != (item = opal_list_remove_first(&ptr->sysinfo))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&ptr->sysinfo);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_nid_t,
opal_object_t,
orte_nid_construct,
orte_nid_destruct);
static void orte_pmap_construct(orte_pmap_t *ptr)
{
ptr->node = -1;
Make the orte_pmap_t an object so it can be properly initialized. Adjust the construct function to properly indicate invalid node/local ranks This commit was SVN r20465.
2009-02-06 18:29:33 +03:00
ptr->local_rank = ORTE_LOCAL_RANK_INVALID;
ptr->node_rank = ORTE_NODE_RANK_INVALID;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_pmap_t,
opal_object_t,
orte_pmap_construct,
NULL);
static void orte_jmap_construct(orte_jmap_t *ptr)
{
ptr->job = ORTE_JOBID_INVALID;
Complete implementation of regular expression generator and parser - now handles leading zero's and suffix in node names. This commit was SVN r21468.
2009-06-18 08:36:00 +04:00
ptr->num_procs = 0;
Fix a race condition that causes corruption of a buffer in mpirun while trying to process launch_local_proc cmds. Cleanup the pidmap handling by changing from value to pointer arrays. This commit was SVN r20629.
2009-02-25 05:43:22 +03:00
OBJ_CONSTRUCT(&ptr->pmap, opal_pointer_array_t);
opal_pointer_array_init(&ptr->pmap,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_jmap_destruct(orte_jmap_t *ptr)
{
Fix a race condition that causes corruption of a buffer in mpirun while trying to process launch_local_proc cmds. Cleanup the pidmap handling by changing from value to pointer arrays. This commit was SVN r20629.
2009-02-25 05:43:22 +03:00
orte_pmap_t **pmaps;
int i;
pmaps = (orte_pmap_t**)ptr->pmap.addr;
for (i=0; i < ptr->pmap.size && NULL != pmaps[i]; i++) {
OBJ_RELEASE(pmaps[i]);
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
OBJ_DESTRUCT(&ptr->pmap);
}
OBJ_CLASS_INSTANCE(orte_jmap_t,
opal_object_t,
orte_jmap_construct,
orte_jmap_destruct);
static void orte_job_map_construct(orte_job_map_t* map)
{
George raised some valid concerns about the extensibility of the revised rmaps framework. Address those by: 1. removing the enum of mapper values 2. change the req_mapper and last_mapper fields to char* so they can hold the component name instead of a mapper flag 3. revise the selection logic in the mapper components to reflect the change. Components now look for their name in the req_mapper field, or to see if other criteria (e.g., npernode) are set that mandate their doing the mapping Several MCA params resided in the rmaps base for historical reasons - they have been in the base since at least the original 1.2 release (and perhaps earlier). However, George correctly pointed out that they really should reside in their respective components. Accordingly, move them to the components, but register synonyms to the old names to avoid breaking backward compatibility. These revisions retain the current functionality of allowing comm_spawn'd jobs to use different mappers than the original job, and for the errmgr to utilize the resilient mapper to recover processes regardless of how they were originally mapped. Given the large number of possible combinations, I am sure that someone will find a corner-case combination of values and selection criteria that cause either no mapper to be selected, or one other than the intended to be used. No one can test all the ways people will use this system, so I expect debugging to continue for awhile. The ability of comm_spawn'd jobs to exploit this functionality relies on changes to the orte_dpm component - this will be committed separately. This commit was SVN r24520.
2011-03-12 08:30:09 +03:00
map->req_mapper = NULL;
map->last_mapper = NULL;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
map->policy = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
map->npernode = 0;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
map->nperboard = 0;
map->npersocket = 0;
map->cpus_per_rank = 1;
map->stride = 1;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
map->oversubscribe = true; /* default to allowing oversubscribe */
map->display_map = false;
map->cpu_lists = false;
map->num_new_daemons = 0;
map->daemon_vpid_start = ORTE_VPID_INVALID;
map->num_nodes = 0;
map->nodes = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(map->nodes,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
}
static void orte_job_map_destruct(orte_job_map_t* map)
{
orte_std_cntr_t i;
George raised some valid concerns about the extensibility of the revised rmaps framework. Address those by: 1. removing the enum of mapper values 2. change the req_mapper and last_mapper fields to char* so they can hold the component name instead of a mapper flag 3. revise the selection logic in the mapper components to reflect the change. Components now look for their name in the req_mapper field, or to see if other criteria (e.g., npernode) are set that mandate their doing the mapping Several MCA params resided in the rmaps base for historical reasons - they have been in the base since at least the original 1.2 release (and perhaps earlier). However, George correctly pointed out that they really should reside in their respective components. Accordingly, move them to the components, but register synonyms to the old names to avoid breaking backward compatibility. These revisions retain the current functionality of allowing comm_spawn'd jobs to use different mappers than the original job, and for the errmgr to utilize the resilient mapper to recover processes regardless of how they were originally mapped. Given the large number of possible combinations, I am sure that someone will find a corner-case combination of values and selection criteria that cause either no mapper to be selected, or one other than the intended to be used. No one can test all the ways people will use this system, so I expect debugging to continue for awhile. The ability of comm_spawn'd jobs to exploit this functionality relies on changes to the orte_dpm component - this will be committed separately. This commit was SVN r24520.
2011-03-12 08:30:09 +03:00
if (NULL != map->req_mapper) {
free(map->req_mapper);
}
if (NULL != map->last_mapper) {
free(map->last_mapper);
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
for (i=0; i < map->nodes->size; i++) {
if (NULL != map->nodes->addr[i]) {
OBJ_RELEASE(map->nodes->addr[i]);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
map->nodes->addr[i] = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
}
OBJ_RELEASE(map->nodes);
}
OBJ_CLASS_INSTANCE(orte_job_map_t,
opal_object_t,
orte_job_map_construct,
orte_job_map_destruct);
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
static void orte_regex_node_construct(orte_regex_node_t *ptr)
{
ptr->prefix = NULL;
Complete implementation of regular expression generator and parser - now handles leading zero's and suffix in node names. This commit was SVN r21468.
2009-06-18 08:36:00 +04:00
OBJ_CONSTRUCT(&ptr->suffix, opal_value_array_t);
opal_value_array_init(&ptr->suffix, sizeof(char));
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
OBJ_CONSTRUCT(&ptr->nodes, opal_value_array_t);
opal_value_array_init(&ptr->nodes, sizeof(int32_t));
OBJ_CONSTRUCT(&ptr->cnt, opal_value_array_t);
opal_value_array_init(&ptr->cnt, sizeof(int32_t));
OBJ_CONSTRUCT(&ptr->starting_vpid, opal_value_array_t);
opal_value_array_init(&ptr->starting_vpid, sizeof(orte_vpid_t));
OBJ_CONSTRUCT(&ptr->ppn, opal_value_array_t);
opal_value_array_init(&ptr->ppn, sizeof(int32_t));
OBJ_CONSTRUCT(&ptr->nrank, opal_value_array_t);
opal_value_array_init(&ptr->nrank, sizeof(orte_node_rank_t));
}
static void orte_regex_node_destruct(orte_regex_node_t *ptr)
{
if (NULL != ptr->prefix) {
free(ptr->prefix);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->prefix = NULL;
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
}
Complete implementation of regular expression generator and parser - now handles leading zero's and suffix in node names. This commit was SVN r21468.
2009-06-18 08:36:00 +04:00
OBJ_DESTRUCT(&ptr->suffix);
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
OBJ_DESTRUCT(&ptr->nodes);
OBJ_DESTRUCT(&ptr->cnt);
OBJ_DESTRUCT(&ptr->starting_vpid);
OBJ_DESTRUCT(&ptr->ppn);
OBJ_DESTRUCT(&ptr->nrank);
}
OBJ_CLASS_INSTANCE(orte_regex_node_t,
opal_list_item_t,
orte_regex_node_construct,
orte_regex_node_destruct);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#endif
Копировать постоянную ссылку