ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
f7f4dd87ab
openmpi/orte/runtime/orte_globals.c

1062 строки
37 KiB
C
Исходник Обычный вид История

Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/*
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* University Research and Technology
* Corporation. All rights reserved.
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
* Copyright (c) 2004-2008 The University of Tennessee and The University
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* of Tennessee Research Foundation. All rights
* reserved.
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
* University of Stuttgart. All rights reserved.
* Copyright (c) 2004-2005 The Regents of the University of California.
* All rights reserved.
A bunch of minor ORTE valgrind-inspired memory leak cleanups (reviewed by Ralph). This commit was SVN r20544.
2009-02-13 07:14:10 +03:00
* Copyright (c) 2007-2009 Cisco Systems, Inc. All rights reserved.
Add the ability to forward SIGTSTP (converted to SIGSTOP) and SIGCONT to the a.outs. By default, they are not forwarded and the behavior remains as it has always been. However, if one runs with --mca orte_forward_job_control 1, then mpirun will catch those two signals and forward them to the orteds which will deliver them to the a.outs. We have had requests for this feature. This commit was SVN r20391.
2009-01-30 21:50:10 +03:00
* Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved.
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "orte_config.h"
#include "orte/constants.h"
#include "orte/types.h"
#ifdef HAVE_SYS_TIME_H
#include <sys/time.h>
#endif
#include "opal/mca/base/mca_base_param.h"
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
#include "opal/mca/paffinity/paffinity.h"
Add missing header file for the full rte case caused by changes in r19471. This commit was SVN r19474. The following SVN revision numbers were found above: r19471 --> open-mpi/ompi@38eb3019195c88797087e99f9456427a020d0402
2008-09-01 21:49:31 +04:00
#include "opal/util/argv.h"
- On the way to get the BTLs split out and lessen dependency on orte: Often, orte/util/show_help.h is included, although no functionality is required -- instead, most often opal_output.h, or orte/mca/rml/rml_types.h Please see orte_show_help_replacement.sh commited next. - Local compilation (Linux/x86_64) w/ -Wimplicit-function-declaration actually showed two *missing* #include "orte/util/show_help.h" in orte/mca/odls/base/odls_base_default_fns.c and in orte/tools/orte-top/orte-top.c Manually added these. Let's have MTT the last word. This commit was SVN r20557.
2009-02-14 05:26:12 +03:00
#include "opal/util/output.h"
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
#include "opal/class/opal_pointer_array.h"
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
#include "opal/class/opal_value_array.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "opal/dss/dss.h"
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
#include "opal/threads/threads.h"
This commit represents a bunch of work on a Mercurial side branch. As such, the commit message back to the master SVN repository is fairly long. = ORTE Job-Level Output Messages = Add two new interfaces that should be used for all new code throughout the ORTE and OMPI layers (we already make the search-and-replace on the existing ORTE / OMPI layers): * orte_output(): (and corresponding friends ORTE_OUTPUT, orte_output_verbose, etc.) This function sends the output directly to the HNP for processing as part of a job-specific output channel. It supports all the same outputs as opal_output() (syslog, file, stdout, stderr), but for stdout/stderr, the output is sent to the HNP for processing and output. More on this below. * orte_show_help(): This function is a drop-in-replacement for opal_show_help(), with two differences in functionality: 1. the rendered text help message output is sent to the HNP for display (rather than outputting directly into the process' stderr stream) 1. the HNP detects duplicate help messages and does not display them (so that you don't see the same error message N times, once from each of your N MPI processes); instead, it counts "new" instances of the help message and displays a message every ~5 seconds when there are new ones ("I got X new copies of the help message...") opal_show_help and opal_output still exist, but they only output in the current process. The intent for the new orte_* functions is that they can apply job-level intelligence to the output. As such, we recommend that all new ORTE and OMPI code use the new orte_* functions, not thei opal_* functions. === New code === For ORTE and OMPI programmers, here's what you need to do differently in new code: * Do not include opal/util/show_help.h or opal/util/output.h. Instead, include orte/util/output.h (this one header file has declarations for both the orte_output() series of functions and orte_show_help()). * Effectively s/opal_output/orte_output/gi throughout your code. Note that orte_output_open() takes a slightly different argument list (as a way to pass data to the filtering stream -- see below), so you if explicitly call opal_output_open(), you'll need to slightly adapt to the new signature of orte_output_open(). * Literally s/opal_show_help/orte_show_help/. The function signature is identical. === Notes === * orte_output'ing to stream 0 will do similar to what opal_output'ing did, so leaving a hard-coded "0" as the first argument is safe. * For systems that do not use ORTE's RML or the HNP, the effect of orte_output_* and orte_show_help will be identical to their opal counterparts (the additional information passed to orte_output_open() will be lost!). Indeed, the orte_* functions simply become trivial wrappers to their opal_* counterparts. Note that we have not tested this; the code is simple but it is quite possible that we mucked something up. = Filter Framework = Messages sent view the new orte_* functions described above and messages output via the IOF on the HNP will now optionally be passed through a new "filter" framework before being output to stdout/stderr. The "filter" OPAL MCA framework is intended to allow preprocessing to messages before they are sent to their final destinations. The first component that was written in the filter framework was to create an XML stream, segregating all the messages into different XML tags, etc. This will allow 3rd party tools to read the stdout/stderr from the HNP and be able to know exactly what each text message is (e.g., a help message, another OMPI infrastructure message, stdout from the user process, stderr from the user process, etc.). Filtering is not active by default. Filter components must be specifically requested, such as: {{{ $ mpirun --mca filter xml ... }}} There can only be one filter component active. = New MCA Parameters = The new functionality described above introduces two new MCA parameters: * '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that help messages will be aggregated, as described above. If set to 0, all help messages will be displayed, even if they are duplicates (i.e., the original behavior). * '''orte_base_show_output_recursions''': An MCA parameter to help debug one of the known issues, described below. It is likely that this MCA parameter will disappear before v1.3 final. = Known Issues = * The XML filter component is not complete. The current output from this component is preliminary and not real XML. A bit more work needs to be done to configure.m4 search for an appropriate XML library/link it in/use it at run time. * There are possible recursion loops in the orte_output() and orte_show_help() functions -- e.g., if RML send calls orte_output() or orte_show_help(). We have some ideas how to fix these, but figured that it was ok to commit before feature freeze with known issues. The code currently contains sub-optimal workarounds so that this will not be a problem, but it would be good to actually solve the problem rather than have hackish workarounds before v1.3 final. This commit was SVN r18434.
2008-05-14 00:00:55 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/mca/errmgr/errmgr.h"
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
#include "orte/mca/rml/rml.h"
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
#include "orte/util/proc_info.h"
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
#include "orte/util/name_fns.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/runtime.h"
Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm. Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed. This commit was SVN r18664.
2008-06-18 07:15:56 +04:00
#include "orte/runtime/runtime_internals.h"
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
#include "orte/runtime/orte_globals.h"
/* need the data type support functions here */
#include "orte/runtime/data_type_support/orte_dt_support.h"
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* globals used by RTE */
bool orte_timing;
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
FILE *orte_timing_output = NULL;
Continue to refine the timing instrumentation to identify where launch time is being spent This commit was SVN r20244.
2009-01-12 22:12:58 +03:00
bool orte_timing_details;
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orte_debug_daemons_file_flag = false;
Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached. Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway. This fixes trac:656 - a REALLY old ticket This commit was SVN r19294. The following Trac tickets were found above: Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 22:59:01 +04:00
bool orte_leave_session_attached;
Implement the seq rmaps module that sequentially maps process ranks to a list hosts in a hostfile. Restore the "do-not-launch" functionality so users can test a mapping without launching it. Add a "do-not-resolve" cmd line flag to mpirun so the opal/util/if.c code does not attempt to resolve network addresses, thus enabling a user to test a hostfile mapping without hanging on network resolve requests. Add a function to hostfile to generate an ordered list of host names from a hostfile This commit was SVN r18190.
2008-04-17 17:50:59 +04:00
bool orte_do_not_launch = false;
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orted_spin_flag = false;
Update the cm ess module to support orted bootstrap. Continue work towards bootstrap capability. This commit was SVN r21989.
2009-09-22 06:16:40 +04:00
bool orte_daemon_bootstrap = false;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
char *orte_local_cpu_model = NULL;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
/* ORTE OOB port flags */
Remove stale define. Add global variable to be used soon. This commit was SVN r18005.
2008-03-28 05:20:37 +03:00
bool orte_static_ports = false;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
char *orte_oob_static_ports = NULL;
When directly launched by an RM, flag that a process is operating without daemons - i.e., standalone. Provide an error string for the new socket_not_available error. Use errmgr.abort to exit when we cannot get a socket, and ensure that the slurmd module returns the proper exit status for slurm 2.0 This commit was SVN r21868.
2009-08-22 06:58:20 +04:00
bool orte_standalone_operation = false;
Provide native integration with SLURM 2.0's OMPI support This commit was SVN r21865.
2009-08-21 22:03:34 +04:00
Cleanup some ignores Add missing variables! This commit was SVN r18063.
2008-04-02 00:32:17 +04:00
bool orte_keep_fqdn_hostnames = false;
Per request from IBM/Eclipse, provide MCA param to request output when nodes are resolved to a different nodename. This really only happens for the node that mpirun executes on, but they need the alert so they can do string matching of node names. This commit was SVN r20032.
2008-11-24 22:57:08 +03:00
bool orte_show_resolved_nodenames;
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
int orted_debug_failure;
int orted_debug_failure_delay;
It is okay for us to init the ORTE mca params multiple times. Indeed, it is absolutely required by orterun as the first time has to be done prior to parsing the command line, which means that the mca values haven't been parsed yet! Add ability for sys admins to prohibit putting session directories under specified locations. Thus, they can now protect parallel file systems from foolish user mistakes. This commit was SVN r18721.
2008-06-24 21:50:56 +04:00
bool orte_homogeneous_nodes = false;
bool orte_hetero_apps = false;
When we hit an error prior to actually launching daemons, it would be nice if orterun didn't bark about daemons failing to launch, mpirun detecting a job failed, etc. Add a new job state to indicate that we never attempted to launch. Flag such a scenario and avoid hitting all the other error messages. This commit was SVN r19366.
2008-08-19 19:19:30 +04:00
bool orte_never_launched = false;
Modify the --display-map and --display-alloc per note on devel list to reduce info for user understanding. Add --display-devel-map and --display-devel-alloc to display all the detailed info we used to provide - it is only of use/interest to developers anyway and confuses users. This commit was SVN r19608.
2008-09-23 19:46:34 +04:00
bool orte_devel_level_output = false;
Fully implement the inbound binomial allgather for daemon-based collectives. Supports both modex and barrier operations. Comm_spawn still uses the rank=0 method - shifting that algo to the daemons is under study. This commit was SVN r18115.
2008-04-10 02:10:53 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
char **orte_launch_environ;
Ensure that we only launch procs on the HNP if that node is actually included in the allocation. This commit was SVN r19038.
2008-07-25 21:13:22 +04:00
bool orte_hnp_is_allocated = false;
Establish an MCA param "orte_allocation_required" so that a system can require the user have an RM-provided allocation in order to run. This helps prevent the problem where a user forgets to get an allocation on an RM-managed cluster, and then executes mpirun on the head node - thus causing all of their mpi procs to launch on the head node, usually bringing it to its knees. Since OMPI allows mpirun to default to the local node, and since users want to retain the option to co-locate procs with mpirun, we needed another param to block this error case. This commit was SVN r19135.
2008-08-04 18:25:19 +04:00
bool orte_allocation_required;
Ensure that we only launch procs on the HNP if that node is actually included in the allocation. This commit was SVN r19038.
2008-07-25 21:13:22 +04:00
A bunch of minor ORTE valgrind-inspired memory leak cleanups (reviewed by Ralph). This commit was SVN r20544.
2009-02-13 07:14:10 +03:00
char *orte_launch_agent = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
char **orted_cmd_line=NULL;
Modify the wakeup system to make it more Windows-friendly. This allows Shiqing to consolidate the Windows-specific modifications into one location, and generalizes the wakeup procedure in case we hit other system-specific requirements. This needs some soak time to ensure we haven't opened any race conditions. I tried to loop everything in the shutdown procedure through that trigger event call to ensure it all goes through the one-time locks as it did before so that someone hitting ctrl-c when we are already shutting down shouldn't cause problems. Just want to let people use it for awhile to verify. This commit was SVN r19159.
2008-08-05 19:09:29 +04:00
Add a new MCA param (and corresponding mpirun cmd line option) to output the debugger proctable info after launch. The output is just the job map with the process pid included, so you get a node-by-node list of the process ranks on that node and thier pids. Works for initial launch and comm_spawn. xml and non-xml output is available This commit was SVN r22725.
2010-02-27 11:32:25 +03:00
/* debugger flags */
Enable co-location of debugger daemons during initial launch and when debugging a running job. Provide support for four MPIR extensions that allow specification of debugger daemon executable, argv for the debugger daemon, whether or not to forward debugger daemon IO, and whether or not debugger daemon will piggy-back on ORTE OOB network. Last is not yet implemented. No change in behavior or operation occurs unless (a) the debugger specifically utilizes the extensions and, for co-locate while running, the user specifically enables the capability via an MCA param. Two of the MPIR extensions supported here are used in a widely-used debugger for a large-scale installation. The other two extensions are new and being utilized in prototype work by several debuggers for possible future release. This commit was SVN r19275.
2008-08-13 21:47:24 +04:00
orte_job_t *orte_debugger_daemon=NULL;
bool orte_enable_debug_cospawn_while_running;
int orte_debugger_check_rate;
Add a new MCA param (and corresponding mpirun cmd line option) to output the debugger proctable info after launch. The output is just the job map with the process pid included, so you get a node-by-node list of the process ranks on that node and thier pids. Works for initial launch and comm_spawn. xml and non-xml output is available This commit was SVN r22725.
2010-02-27 11:32:25 +03:00
bool orte_output_debugger_proctable=false;
Cleanup the debugger daemon co-launch code and add an ability to test it. Implement ability to co-launch debugger daemons upon attach to a running job for jobs launched under rsh, slurm, and tm environments (others can easily be added if desired). Add new mca params to test: orte_debugger_test_daemon: Name of the executable to be used to simulate a debugger colaunch orte_debugger_test_attach: Test debugger colaunch after debugger attachment To test co-launch at job start, just set the orte_debugger_test_daemon param. To test co-launch upon attach: set orte_debugger_test_daemon set orte_debugger_test_attach=1 set orte_enable_debug_cospawn_while_running=1 set orte_debugger_check_rate=<N> - defines the number of seconds to wait before "checking" for a debugger attaching Added a "debugger" program to orte/test/mpi that just spins to simulate a debugger daemon. This commit was SVN r23144.
2010-05-14 22:44:49 +04:00
char *orte_debugger_test_daemon=NULL;
bool orte_debugger_test_attach=false;
Enable co-location of debugger daemons during initial launch and when debugging a running job. Provide support for four MPIR extensions that allow specification of debugger daemon executable, argv for the debugger daemon, whether or not to forward debugger daemon IO, and whether or not debugger daemon will piggy-back on ORTE OOB network. Last is not yet implemented. No change in behavior or operation occurs unless (a) the debugger specifically utilizes the extensions and, for co-locate while running, the user specifically enables the capability via an MCA param. Two of the MPIR extensions supported here are used in a widely-used debugger for a large-scale installation. The other two extensions are new and being utilized in prototype work by several debuggers for possible future release. This commit was SVN r19275.
2008-08-13 21:47:24 +04:00
Cleanup the debugger daemon co-launch code and add an ability to test it. Implement ability to co-launch debugger daemons upon attach to a running job for jobs launched under rsh, slurm, and tm environments (others can easily be added if desired). Add new mca params to test: orte_debugger_test_daemon: Name of the executable to be used to simulate a debugger colaunch orte_debugger_test_attach: Test debugger colaunch after debugger attachment To test co-launch at job start, just set the orte_debugger_test_daemon param. To test co-launch upon attach: set orte_debugger_test_daemon set orte_debugger_test_attach=1 set orte_enable_debug_cospawn_while_running=1 set orte_debugger_check_rate=<N> - defines the number of seconds to wait before "checking" for a debugger attaching Added a "debugger" program to orte/test/mpi that just spins to simulate a debugger daemon. This commit was SVN r23144.
2010-05-14 22:44:49 +04:00
/* exit triggers and flags */
Modify the wakeup system to make it more Windows-friendly. This allows Shiqing to consolidate the Windows-specific modifications into one location, and generalizes the wakeup procedure in case we hit other system-specific requirements. This needs some soak time to ensure we haven't opened any race conditions. I tried to loop everything in the shutdown procedure through that trigger event call to ensure it all goes through the one-time locks as it did before so that someone hitting ctrl-c when we are already shutting down shouldn't cause problems. Just want to let people use it for awhile to verify. This commit was SVN r19159.
2008-08-05 19:09:29 +04:00
orte_trigger_event_t orte_exit, orteds_exit;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
int orte_exit_status = 0;
bool orte_abnormal_term_ordered = false;
This is a first step towards supporting fully-routed OOB communications: 1. remove direct routed module (hooray!) 2. add radix tree routed module (binomial remains default) 3. remove duplicate data storage - orteds were storing nidmap and pidmap data in odls, everyone else in ess 4. add ess APIs to update nidmap, add new pidmap - used only by orteds for MPI-2 support 5. modify code to eliminate multiple calls to orte_routed.update_route that recreated info already in ess pidmap. Add ess API to lookup that info instead. Modify routed modules to utilize that capability 6. setup new ability to shutdown orteds without sending back an "ack" message to mpirun - not utilized yet, will require some changes to plm terminate_orteds functions in managed environments (coming soon) Initial tests indicating that fully routing comm via defined routing trees may not actually have a significant cost for operations like IB QP setup. More tests required to confirm. This will require an autogen... This commit was SVN r19866.
2008-11-01 00:10:00 +03:00
bool orte_routing_is_enabled = false;
Separate abnormal job termination from abnormal orted termination so we can continue to use xcast for orted cmds, but can know to turn off reading of stdin as the job is being terminated. This commit was SVN r20650.
2009-02-27 13:16:25 +03:00
bool orte_job_term_ordered = false;
Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time. Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher. Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported. More info to come on devel list. This commit was SVN r18555.
2008-06-03 01:46:34 +04:00
int orte_startup_timeout;
Fully implement the inbound binomial allgather for daemon-based collectives. Supports both modex and barrier operations. Comm_spawn still uses the rank=0 method - shifting that algo to the daemons is under study. This commit was SVN r18115.
2008-04-10 02:10:53 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
int orte_timeout_usec_per_proc;
float orte_max_timeout;
Fix tree-spawn to work within the new modex system This commit was SVN r18349.
2008-05-01 23:19:34 +04:00
opal_buffer_t *orte_tree_launch_cmd = NULL;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* global arrays for data storage */
Replace all occurences of orte_pointer_array by opal_pointer_array. Remove the implementation of orte_pointer_array. This commit was SVN r17636.
2008-02-28 08:32:23 +03:00
opal_pointer_array_t *orte_job_data;
opal_pointer_array_t *orte_node_pool;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Cleanup the output for nodename resolve reporting This commit was SVN r20081.
2008-12-08 22:00:36 +03:00
/* a clean output channel without prefix */
int orte_clean_output = -1;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
/* RHC: FLAG TO SELECT WHETHER OR NOT TO SEND PROFILE FILE IN NIDMAP */
bool orte_send_profile;
/* Nidmap and job maps */
opal_pointer_array_t orte_nidmap;
opal_pointer_array_t orte_jobmap;
Continue development of regular expression support by implementing it for slurm launches. Works for both initial (cmd line and non-cmd line) and comm_spawn launch. Additional work required to fully enable static port support when using cmd line regular expression launch system. This commit was SVN r21502.
2009-06-24 00:25:38 +04:00
bool orte_use_regexp;
char *orted_launch_cmd = NULL;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
/* list of local children on a daemon */
opal_list_t orte_local_children;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
opal_mutex_t orte_local_children_lock;
opal_condition_t orte_local_children_cond;
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
/* list of job data for local children on a daemon */
opal_list_t orte_local_jobdata;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
opal_mutex_t orte_local_jobdata_lock;
opal_condition_t orte_local_jobdata_cond;
Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately. This commit was SVN r20228.
2009-01-08 17:25:56 +03:00
Fix a race condition in the IOF and add some new user-requested features: 1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen. 2. add a new "timestamp-output" option that timestamp's each line of output 3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file. 4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window. This commit was SVN r20392.
2009-01-31 01:47:30 +03:00
/* IOF controls */
bool orte_tag_output;
bool orte_timestamp_output;
char *orte_output_filename;
/* generate new xterm windows to display output from specified ranks */
char *orte_xterm;
Add the ability to forward SIGTSTP (converted to SIGSTOP) and SIGCONT to the a.outs. By default, they are not forwarded and the behavior remains as it has always been. However, if one runs with --mca orte_forward_job_control 1, then mpirun will catch those two signals and forward them to the orteds which will deliver them to the a.outs. We have had requests for this feature. This commit was SVN r20391.
2009-01-30 21:50:10 +03:00
/* whether or not to forward SIGTSTP and SIGCONT signals */
bool orte_forward_job_control;
Enable comm_spawn of slave processes, currently only active for the rsh, slurm, and tm environments. Establish support for local rsh environments in the plm/base so that rsh of local slaves can be done by any environment that supports it. Create new orte_rsh_agent param so users can specify rsh agent from outside of rsh plm, and sym link that to the old plm_rsh_agent and pls_rsh_agent options. Modify the orte-bootproxy to pass prefix for the remote slave to support hetero/hybrid scenarios This commit was SVN r20492.
2009-02-09 23:44:44 +03:00
/* rsh support */
A bunch of minor ORTE valgrind-inspired memory leak cleanups (reviewed by Ralph). This commit was SVN r20544.
2009-02-13 07:14:10 +03:00
char *orte_rsh_agent = NULL;
Add the ability to directly launch procs via rsh/ssh. Collect common functions in plm/base. Create a new global param to set assume_same_shell, alias'd back to plm_rsh_assume_same_shell (not deprecated). This commit was SVN r21328.
2009-05-30 05:10:25 +04:00
bool orte_assume_same_shell = true;
Enable comm_spawn of slave processes, currently only active for the rsh, slurm, and tm environments. Establish support for local rsh environments in the plm/base so that rsh of local slaves can be done by any environment that supports it. Create new orte_rsh_agent param so users can specify rsh agent from outside of rsh plm, and sym link that to the old plm_rsh_agent and pls_rsh_agent options. Modify the orte-bootproxy to pass prefix for the remote slave to support hetero/hybrid scenarios This commit was SVN r20492.
2009-02-09 23:44:44 +03:00
Some cleanups required from last night's commits to resolve some race conditions and ensure we cleanup properly. Also, remove some debug output that was unintentionally left "on" by default. This commit was SVN r21202.
2009-05-11 18:03:07 +04:00
/* orted exit with barrier */
bool orte_orted_exit_with_barrier = true;
Provide a "progress meter" for launch that outputs progress as we are launching, especially on large jobs. Also, provide a timeout mechanism so that we cleanly abort if we don't get a response from the next daemon in a specified time. This commit was SVN r21359.
2009-06-03 03:52:02 +04:00
/* report launch progress */
bool orte_report_launch_progress = false;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
/* cluster hardware info */
uint8_t orte_default_num_boards;
uint8_t orte_default_num_sockets_per_board;
uint8_t orte_default_num_cores_per_socket;
/* allocation specification */
char *orte_default_cpu_set;
Correct a couple of bugs in the rank_file mapper that were incorrectly assigning vpids. Add a capability to parse the rankfile to extract node information in place of requiring both hostfile and rankfile for non-RM managed environments. The rankfile is -only- parsed for this IF the hostfile and -host options are not given. Otherwise, those are used to establish allocation info as we did before this commit. This commit was SVN r21815.
2009-08-13 20:08:43 +04:00
char *orte_default_hostfile = NULL;
char *orte_rankfile;
Use this option only on Windows, so protect it with #ifdef __WINDOWS__. This commit was SVN r22701.
2010-02-24 11:50:03 +03:00
#ifdef __WINDOWS__
Add a global option for inputing the head node name for Windows CCP trough command line. This commit was SVN r22688.
2010-02-23 22:42:51 +03:00
char *orte_ccp_headnode;
Use this option only on Windows, so protect it with #ifdef __WINDOWS__. This commit was SVN r22701.
2010-02-24 11:50:03 +03:00
#endif
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
/* default rank assigment and binding policy */
orte_mapping_policy_t orte_default_mapping_policy = 0;
Enable a new command line option to --report-events that instructs mpirun to RML-report specific events during job life to the requestor. This commit was SVN r21954.
2009-09-09 09:28:45 +04:00
/* tool communication controls */
bool orte_report_events = false;
char *orte_report_events_uri = NULL;
Instead of an odls-base mca param, make report_bindings a global param so that we can (a) detect it was set in the plm, and then (b) ensure it gets passed along to remote orteds so they will comply with the request. This commit was SVN r22021.
2009-09-28 07:17:15 +04:00
/* report bindings */
bool orte_report_bindings = false;
The ability to add procs to a running job was unfortunately borked when we added the detection of a proc exiting before calling init. Re-enable it here, ensuring that procs that are being restarted and/or added to a job do -not- call barrier during orte_init. This commit was SVN r22404.
2010-01-14 20:59:42 +03:00
/* barrier control */
bool orte_do_not_barrier = false;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
/* process recovery */
bool orte_enable_recovery;
int32_t orte_max_global_restarts;
int32_t orte_max_local_restarts;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
/* comm fn for updating state */
orte_default_comm_fn_t orte_comm;
Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases. Implement process migration when failed nodes are detected. Some testing still required This commit was SVN r23121.
2010-05-12 22:11:58 +04:00
/* exit status reporting */
bool orte_report_child_jobs_separately;
struct timeval orte_child_time_to_exit;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#endif /* !ORTE_DISABLE_FULL_RTE */
Missed a symbol that needs to exist in non-full RTE case This commit was SVN r19482.
2008-09-02 19:07:48 +04:00
int orte_debug_output = -1;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
bool orte_debug_daemons_flag = false;
bool orte_xml_output = false;
Update the xml support to allow -xml-file foo whereby we redirect all xml formatted output (and ONLY xml formatted output) to a specified file This commit was SVN r21930.
2009-09-02 22:03:10 +04:00
FILE *orte_xml_fp = NULL;
Use orte_notifier to alert sys admins to checksum violations in the csum pml. Add ability to store the RM's jobid string to tag the notifier message so that the sys admin knows what job had the problem. This commit was SVN r21687.
2009-07-15 23:43:26 +04:00
char *orte_job_ident = NULL;
Truly respect the -quiet option. Make it an mca param so someone doesn't have to put it solely on the cmd line. Tell show_help to shaddup as well. This commit was SVN r22926.
2010-04-02 18:19:38 +04:00
bool orte_execute_quiet = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Fixes trac:1361: mainly add new internal MCA parameter that orterun will set when it launches under debuggers using the --debug option. This commit was SVN r19116. The following Trac tickets were found above: Ticket 1361 --> https://svn.open-mpi.org/trac/ompi/ticket/1361
2008-08-01 02:11:46 +04:00
/* See comment in orte/tools/orterun/debuggers.c about this MCA
param */
bool orte_in_parallel_debugger = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
int orte_dt_init(void)
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
{
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
int rc;
opal_data_type_t tmp;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* set default output */
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
orte_debug_output = opal_output_open(NULL);
Cleanup the output for nodename resolve reporting This commit was SVN r20081.
2008-12-08 22:00:36 +03:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/* open up the verbose output for ORTE debugging */
if (orte_debug_flag || 0 < orte_debug_verbosity ||
Modify the orte_process_info structure to handle a broader range of process types by replacing the individual booleans with a 32-bit bitmap. Use a set of #define's to define the individual bits, and a set of matching macros to test for them. Update the orte code base to use the macros instead of the booleans. Minor mod to the ompi layer to use the new #define's - just one-line name replacements. This commit was SVN r21144.
2009-05-04 15:07:40 +04:00
(orte_debug_daemons_flag && (ORTE_PROC_IS_DAEMON || ORTE_PROC_IS_HNP))) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
if (0 < orte_debug_verbosity) {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output_set_verbosity(orte_debug_output, orte_debug_verbosity);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
} else {
Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP. After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach. I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive. This commit was SVN r18619.
2008-06-09 18:53:58 +04:00
opal_output_set_verbosity(orte_debug_output, 1);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
}
}
ORTE-level MCA params are defined in several places. Ompi_info cannot call orte_init due to an issue with the memory allocator, thus making it impossible for ompi_info to display all of the ORTE-level MCA params. By consolidating them all into one function, ompi_info can call that function and register the desired variables. This also requires, however, that ompi_info call orte_output_init to avoid generating tons of error messages, so make that adjustment too. Fixes ticket #1314 In addition, orte_output has a race condition issue whereby calls to orte_output/verbose can occur prior to either the RML being defined/setup, or the HNP being defined. This latter occurs during the initialization of the orte_process_info structure. In both cases, there is no way orte_output can send the output to the HNP. Hence, the message must be simply output locally. Fixes ticket #1315 This commit was SVN r18524.
2008-05-28 17:29:58 +04:00
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
/** register the base system types with the DSS */
tmp = ORTE_STD_CNTR;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_std_cntr,
orte_dt_unpack_std_cntr,
(opal_dss_copy_fn_t)orte_dt_copy_std_cntr,
(opal_dss_compare_fn_t)orte_dt_compare_std_cntr,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_STD_CNTR", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NAME;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_name,
orte_dt_unpack_name,
(opal_dss_copy_fn_t)orte_dt_copy_name,
(opal_dss_compare_fn_t)orte_dt_compare_name,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_print_name,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_NAME", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_VPID;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_vpid,
orte_dt_unpack_vpid,
(opal_dss_copy_fn_t)orte_dt_copy_vpid,
(opal_dss_compare_fn_t)orte_dt_compare_vpid,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_VPID", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOBID;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_jobid,
orte_dt_unpack_jobid,
(opal_dss_copy_fn_t)orte_dt_copy_jobid,
(opal_dss_compare_fn_t)orte_dt_compare_jobid,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_JOBID", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
tmp = ORTE_JOB;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_job,
orte_dt_unpack_job,
(opal_dss_copy_fn_t)orte_dt_copy_job,
(opal_dss_compare_fn_t)orte_dt_compare_job,
(opal_dss_size_fn_t)orte_dt_size_job,
(opal_dss_print_fn_t)orte_dt_print_job,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_JOB", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_node,
orte_dt_unpack_node,
(opal_dss_copy_fn_t)orte_dt_copy_node,
(opal_dss_compare_fn_t)orte_dt_compare_node,
(opal_dss_size_fn_t)orte_dt_size_node,
(opal_dss_print_fn_t)orte_dt_print_node,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_NODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_PROC;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_proc,
orte_dt_unpack_proc,
(opal_dss_copy_fn_t)orte_dt_copy_proc,
(opal_dss_compare_fn_t)orte_dt_compare_proc,
(opal_dss_size_fn_t)orte_dt_size_proc,
(opal_dss_print_fn_t)orte_dt_print_proc,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_PROC", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_APP_CONTEXT;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_app_context,
orte_dt_unpack_app_context,
(opal_dss_copy_fn_t)orte_dt_copy_app_context,
(opal_dss_compare_fn_t)orte_dt_compare_app_context,
(opal_dss_size_fn_t)orte_dt_size_app_context,
(opal_dss_print_fn_t)orte_dt_print_app_context,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_APP_CONTEXT", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_NODE_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_node_state,
orte_dt_unpack_node_state,
(opal_dss_copy_fn_t)orte_dt_copy_node_state,
(opal_dss_compare_fn_t)orte_dt_compare_node_state,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_NODE_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_PROC_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_proc_state,
orte_dt_unpack_proc_state,
(opal_dss_copy_fn_t)orte_dt_copy_proc_state,
(opal_dss_compare_fn_t)orte_dt_compare_proc_state,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_PROC_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOB_STATE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_job_state,
orte_dt_unpack_job_state,
(opal_dss_copy_fn_t)orte_dt_copy_job_state,
(opal_dss_compare_fn_t)orte_dt_compare_job_state,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_JOB_STATE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_EXIT_CODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_exit_code,
orte_dt_unpack_exit_code,
(opal_dss_copy_fn_t)orte_dt_copy_exit_code,
(opal_dss_compare_fn_t)orte_dt_compare_exit_code,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_EXIT_CODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_JOB_MAP;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_map,
orte_dt_unpack_map,
(opal_dss_copy_fn_t)orte_dt_copy_map,
(opal_dss_compare_fn_t)orte_dt_compare_map,
(opal_dss_size_fn_t)orte_dt_size_map,
(opal_dss_print_fn_t)orte_dt_print_map,
(opal_dss_release_fn_t)orte_dt_std_obj_release,
OPAL_DSS_STRUCTURED,
"ORTE_JOB_MAP", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_RML_TAG;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_tag,
orte_dt_unpack_tag,
(opal_dss_copy_fn_t)orte_dt_copy_tag,
(opal_dss_compare_fn_t)orte_dt_compare_tags,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_RML_TAG", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_DAEMON_CMD;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_daemon_cmd,
orte_dt_unpack_daemon_cmd,
(opal_dss_copy_fn_t)orte_dt_copy_daemon_cmd,
(opal_dss_compare_fn_t)orte_dt_compare_daemon_cmd,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_DAEMON_CMD", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
tmp = ORTE_GRPCOMM_MODE;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_grpcomm_mode,
orte_dt_unpack_grpcomm_mode,
(opal_dss_copy_fn_t)orte_dt_copy_grpcomm_mode,
(opal_dss_compare_fn_t)orte_dt_compare_grpcomm_mode,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_GRPCOMM_MODE", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
tmp = ORTE_IOF_TAG;
if (ORTE_SUCCESS != (rc = opal_dss.register_type(orte_dt_pack_iof_tag,
orte_dt_unpack_iof_tag,
(opal_dss_copy_fn_t)orte_dt_copy_iof_tag,
(opal_dss_compare_fn_t)orte_dt_compare_iof_tag,
(opal_dss_size_fn_t)orte_dt_std_size,
(opal_dss_print_fn_t)orte_dt_std_print,
(opal_dss_release_fn_t)orte_dt_std_release,
OPAL_DSS_UNSTRUCTURED,
"ORTE_IOF_TAG", &tmp))) {
ORTE_ERROR_LOG(rc);
return rc;
}
Number of small fixes to get the trunk to build again on Catamount This commit was SVN r20141.
2008-12-16 23:09:56 +03:00
#endif /* !ORTE_DISABLE_FULL_SUPPORT */
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return ORTE_SUCCESS;
}
Some fixes for the disabled ORTE case * Protect an orte variable used in the orte debugger stuff * Initialize the datatype code in the Catamount code, as we need it for intercommunicators (the proc code needs it to pack the remote name) * Turn on a bunch of the orte datatype code so that ORTE_NAME is available. This commit was SVN r19457.
2008-08-31 22:06:55 +04:00
#if !ORTE_DISABLE_FULL_SUPPORT
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
orte_job_t* orte_get_job_data_object(orte_jobid_t job)
{
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
int32_t ljob;
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
Track process heartbeats with time_t, be a little less restrictive on who can retrieve an orte_job_t object This commit was SVN r22921.
2010-03-31 23:20:06 +04:00
/* if the job data wasn't setup, we cannot provide the data */
if (NULL == orte_job_data) {
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
return NULL;
}
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
/* the job is indexed by its local jobid, so we can
* just look it up here. it is not an error for this
* to not be found - could just be
Eliminate an annoying "not found" message when a job abnormally terminates. In this case, we can get a race condition where the job object has been removed, but updates are continuing to flow into the system. This commit was SVN r20857.
2009-03-24 21:06:49 +03:00
* a race condition whereby the job has already been
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
* removed from the array. The get_item function
* will just return NULL in that case.
Eliminate an annoying "not found" message when a job abnormally terminates. In this case, we can get a race condition where the job object has been removed, but updates are continuing to flow into the system. This commit was SVN r20857.
2009-03-24 21:06:49 +03:00
*/
More cleanup of pointer array usage This commit was SVN r20981.
2009-04-13 23:06:54 +04:00
ljob = ORTE_LOCAL_JOBID(job);
return (orte_job_t*)opal_pointer_array_get_item(orte_job_data, ljob);
Merge the ORTE devel branch into the main trunk. Details of what this means will be circulated separately. Remains to be tested to ensure everything came over cleanly, so please continue to withhold commits a little longer This commit was SVN r17632.
2008-02-28 04:57:57 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
int orte_global_comm(orte_process_name_t *recipient,
opal_buffer_t *buf, orte_rml_tag_t tag,
orte_default_cbfunc_t cbfunc)
{
int ret;
if (recipient->jobid == ORTE_PROC_MY_NAME->jobid &&
recipient->vpid == ORTE_PROC_MY_NAME->vpid &&
NULL != cbfunc) {
/* if I am the recipient and a direct fn is provided, use a message event */
ORTE_MESSAGE_EVENT(ORTE_PROC_MY_NAME, buf, tag, cbfunc);
ret = ORTE_SUCCESS;
} else {
/* go ahead and send it */
if (0 > (ret = orte_rml.send_buffer(recipient, buf, tag, 0))) {
ORTE_ERROR_LOG(ret);
} else {
ret = ORTE_SUCCESS;
}
}
return ret;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
/*
* CONSTRUCTORS, DESTRUCTORS, AND CLASS INSTANTIATIONS
* FOR ORTE CLASSES
*/
static void orte_app_context_construct(orte_app_context_t* app_context)
{
app_context->idx=0;
app_context->app=NULL;
app_context->num_procs=0;
app_context->argv=NULL;
app_context->env=NULL;
app_context->cwd=NULL;
app_context->user_specified_cwd=false;
app_context->hostfile=NULL;
app_context->add_hostfile=NULL;
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
app_context->add_host = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
app_context->dash_host = NULL;
app_context->prefix_dir = NULL;
app_context->preload_binary = false;
app_context->preload_files = NULL;
app_context->preload_files_dest_dir = NULL;
Add ability for local slave spawns to pre-position supporting files. Update comm_spawn and comm_spawn_multiple man pages to cover new info_keys. This commit was SVN r20527.
2009-02-12 18:56:45 +03:00
app_context->preload_files_src_dir = NULL;
Improve the efficiency of the launch system by changing the outer loop to being over app_context, and adding a flag to the app_context so the daemon can record that "this app is on my node" when decoding the launch msg. If the --wdir option is given, check to see if the user provided a relative path. If so, convert it to an absolute path. This is needed to maintain consistent behavior across environements. Some environments automatically chdir to your current working directory when launching the remote orted, while others (e.g., ssh) don't. This levels the playing field and reduces user surprise. This commit was SVN r20342.
2009-01-25 15:39:24 +03:00
app_context->used_on_node = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_app_context_destructor(orte_app_context_t* app_context)
{
if (NULL != app_context->app) {
free (app_context->app);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->app = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
/* argv and env lists created by util/argv copy functions */
if (NULL != app_context->argv) {
opal_argv_free(app_context->argv);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->argv = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->env) {
opal_argv_free(app_context->env);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->env = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->cwd) {
free (app_context->cwd);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->cwd = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->hostfile) {
free(app_context->hostfile);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->hostfile = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->add_hostfile) {
free(app_context->add_hostfile);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->add_hostfile = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
if (NULL != app_context->add_host) {
opal_argv_free(app_context->add_host);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->add_host = NULL;
Add support for the add-host and add-hostfile MPI Info keys to allow Comm_spawn users to add new hosts to those already known by mpirun. Requires full testing once comm_spawn is fixed (Edgar is working that now). This commit was SVN r21664.
2009-07-14 18:34:11 +04:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != app_context->dash_host) {
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
opal_argv_free(app_context->dash_host);
app_context->dash_host = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != app_context->prefix_dir) {
free(app_context->prefix_dir);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->prefix_dir = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
app_context->preload_binary = false;
Add a new MPI_Info key to preposition OMPI libraries - implementation underway, but this just defines and passes the new key This commit was SVN r21425.
2009-06-12 21:53:13 +04:00
app_context->preload_libs = false;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if(NULL != app_context->preload_files) {
free(app_context->preload_files);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->preload_files = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if(NULL != app_context->preload_files_dest_dir) {
free(app_context->preload_files_dest_dir);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->preload_files_dest_dir = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add ability for local slave spawns to pre-position supporting files. Update comm_spawn and comm_spawn_multiple man pages to cover new info_keys. This commit was SVN r20527.
2009-02-12 18:56:45 +03:00
if(NULL != app_context->preload_files_src_dir) {
free(app_context->preload_files_src_dir);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
app_context->preload_files_src_dir = NULL;
Add ability for local slave spawns to pre-position supporting files. Update comm_spawn and comm_spawn_multiple man pages to cover new info_keys. This commit was SVN r20527.
2009-02-12 18:56:45 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_app_context_t,
opal_object_t,
orte_app_context_construct,
orte_app_context_destructor);
static void orte_job_construct(orte_job_t* job)
{
job->jobid = ORTE_JOBID_INVALID;
job->apps = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(job->apps,
1,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
2);
job->num_apps = 0;
Default job controls to forward IO. Adjust debugger code to not forward IO unless requested. This commit was SVN r19690.
2008-10-06 21:56:23 +04:00
job->controls = ORTE_JOB_CONTROL_FORWARD_OUTPUT;
Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code: 1. completely and cleanly separates responsibilities between the HNP, orted, and tool components. 2. removes all wireup messaging during launch and shutdown. 3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol. 4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0. 5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none". 6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout. 7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output" This is not intended for the 1.3 release as it is a major change requiring considerable soak time. This commit was SVN r19767.
2008-10-18 04:00:49 +04:00
job->stdin_target = ORTE_VPID_INVALID;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->total_slots_alloc = 0;
job->num_procs = 0;
job->procs = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(job->procs,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
job->map = NULL;
job->bookmark = NULL;
job->oversubscribe_override = false;
job->state = ORTE_JOB_STATE_UNDEF;
job->num_launched = 0;
job->num_reported = 0;
job->num_terminated = 0;
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
job->num_daemons_reported = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->abort = false;
job->aborted_proc = NULL;
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
OBJ_CONSTRUCT(&job->reported_lock, opal_mutex_t);
OBJ_CONSTRUCT(&job->reported_cond, opal_condition_t);
job->not_reported = true;
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params: 1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis 2. orte_max_local_restarts - default max number of local restarts, can be overridden 3. orte_max_global_restarts - default max number of relocates, can be overridden Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code. This commit was SVN r23057.
2010-04-28 08:06:57 +04:00
job->enable_recovery = false;
job->max_local_restarts = -1;
job->max_global_restarts = -1;
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
job->launch_msg_sent.tv_sec = 0;
job->launch_msg_sent.tv_usec = 0;
job->max_launch_msg_recvd.tv_sec = 0;
job->max_launch_msg_recvd.tv_usec = 0;
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
job->ckpt_state = 0;
job->ckpt_snapshot_ref = NULL;
job->ckpt_snapshot_loc = NULL;
#endif
}
static void orte_job_destruct(orte_job_t* job)
{
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
orte_proc_t *proc;
orte_app_context_t *app;
orte_job_t *jdata;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
int n;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Some cleanups required from last night's commits to resolve some race conditions and ensure we cleanup properly. Also, remove some debug output that was unintentionally left "on" by default. This commit was SVN r21202.
2009-05-11 18:03:07 +04:00
if (NULL == job) {
/* probably just a race condition - just return */
return;
}
if (orte_debug_flag) {
opal_output(0, "%s Releasing job data for %s",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), ORTE_JOBID_PRINT(job->jobid));
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
for (n=0; n < job->apps->size; n++) {
if (NULL == (app = (orte_app_context_t*)opal_pointer_array_get_item(job->apps, n))) {
continue;
}
OBJ_RELEASE(app);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(job->apps);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != job->map) {
OBJ_RELEASE(job->map);
job->map = NULL;
}
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
for (n=0; n < job->procs->size; n++) {
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
if (NULL == (proc = (orte_proc_t*)opal_pointer_array_get_item(job->procs, n))) {
continue;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
OBJ_RELEASE(proc);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(job->procs);
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base: * add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway. * update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states. * remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails. * update the plm_base_receive.c code to call the errmgr with state updates from remote daemons * update the routed modules to reflect that process state is updated in the errmgr * ensure that the orted's open the errmgr and select their appropriate module * add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place * define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired. * update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios This commit was SVN r23023.
2010-04-23 08:44:41 +04:00
OBJ_DESTRUCT(&job->reported_lock);
OBJ_DESTRUCT(&job->reported_cond);
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != job->ckpt_snapshot_ref) {
free(job->ckpt_snapshot_ref);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
job->ckpt_snapshot_ref = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != job->ckpt_snapshot_loc) {
free(job->ckpt_snapshot_loc);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
job->ckpt_snapshot_loc = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
#endif
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
/* find the job in the global array */
Some cleanups required from last night's commits to resolve some race conditions and ensure we cleanup properly. Also, remove some debug output that was unintentionally left "on" by default. This commit was SVN r21202.
2009-05-11 18:03:07 +04:00
if (NULL != orte_job_data) {
for (n=0; n < orte_job_data->size; n++) {
if (NULL == (jdata = (orte_job_t*)opal_pointer_array_get_item(orte_job_data, n))) {
continue;
}
if (jdata->jobid == job->jobid) {
/* set the entry to NULL */
opal_pointer_array_set_item(orte_job_data, n, NULL);
break;
}
The current errmgr.register_callback API takes a jobid as one of its argument. The intent was to have the errmgr check the jobid of the job being reported to it and, if it matches the jobid that was registered, call the specified callback function. Unfortunately, we assign the jobid during the plm.spawn procedure - which means it happens -after- control of the job has passed out of the range of mpirun (or whatever program is spawning the job), so it is too late for that main program to register a callback function. If the main program registers tha callback -after- we return from plm.spawn, then it (a) cannot get a callback for failed-to-start, and (b) will miss the callback if a proc aborts in the time between job launch and the call to errmgr.register_callback. This commit fixes the problem by adding callback-related fields to the orte_job_t object. Thus, the main program can specify what job states should initiate a callback, what function is to be called, and what data is to be passed back by simply filling in the orte_job_t fields prior to calling plm.spawn. Also, fully implement the "copy" function for the orte_job_t object. NOTE: as a result of this change, the errmgr.register_callback API may no longer be of any value. This commit was SVN r21200.
2009-05-11 07:38:15 +04:00
}
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_job_t,
opal_list_item_t,
orte_job_construct,
orte_job_destruct);
static void orte_node_construct(orte_node_t* node)
{
node->name = NULL;
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
node->alias = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->index = -1;
node->daemon = NULL;
node->daemon_launched = false;
node->launch_id = -1;
node->num_procs = 0;
node->procs = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(node->procs,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
node->next_node_rank = 0;
node->oversubscribed = false;
node->state = ORTE_NODE_STATE_UNKNOWN;
node->slots = 0;
node->slots_inuse = 0;
node->slots_alloc = 0;
node->slots_max = 0;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
node->boards = orte_default_num_boards;
node->sockets_per_board = orte_default_num_sockets_per_board;
node->cores_per_socket = orte_default_num_cores_per_socket;
if (NULL != orte_default_cpu_set) {
node->cpu_set = strdup(orte_default_cpu_set);
} else {
node->cpu_set = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
node->username = NULL;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
OBJ_CONSTRUCT(&node->resources, opal_list_t);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_node_destruct(orte_node_t* node)
{
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
int i;
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
opal_list_item_t *item;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != node->name) {
free(node->name);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->name = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
if (NULL != node->alias) {
opal_argv_free(node->alias);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->alias = NULL;
Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option. This commit was SVN r20284.
2009-01-15 21:11:50 +03:00
}
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
if (NULL != node->daemon) {
node->daemon->node = NULL;
OBJ_RELEASE(node->daemon);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->daemon = NULL;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
for (i=0; i < node->procs->size; i++) {
if (NULL != node->procs->addr[i]) {
((orte_proc_t*)(node->procs->addr[i]))->node = NULL;
OBJ_RELEASE(node->procs->addr[i]);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->procs->addr[i] = NULL;
Modify the accounting system to recycle jobids. Properly recover resources from nodes and jobs upon completion. Adjustments in several places were required to deal with sparsely populated job, node, and proc arrays as a result of this change. Correct an error wrt how jobids were being computed. Needed to ensure that the job family field was not overrun as we increment jobids for comm_spawn. Update the slurm plm module so it uses the new slurm termination procedure (brings trunk back into alignment with 1.3 branch). Update the slurmd ess component so it doesn't get selected if we are running a singleton inside of a slurm allocation. Cleanup HNP init by moving some code that had been in orte_globals.c for historical reasons into the ess hnp module, and removing the call to that code from the ess_base_std_prolog NOTE: this change allows orte to support an infinite aggregate number of comm_spawn's, with up to 64k being alive at any one instant. HOWEVER, the MPI layer currently does -not- support re-use of jobids. I did some prototype coding to revise the ompi_proc_t structures, but the BTLs are caching their own data, and there was no readily apparent way to update it. Thus, attempts to spawn more than the 64k limit will abort to avoid causing the MPI layer to hang. This commit was SVN r20700.
2009-03-03 19:39:13 +03:00
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_RELEASE(node->procs);
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
if (NULL != node->cpu_set) {
free(node->cpu_set);
node->cpu_set = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != node->username) {
free(node->username);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
node->username = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Add a new framework for discovering local resource information such as cpu type/model, #cpus, available physical memory, etc. Two initial components (darwin and linux) are provided. This is needed to support bootstrap operations where daemons are started at node boot, and applications where initial knowledge of cpu identification is needed to guide framework component selection. Add orte configuration option to control the use of the framework in the system. Although the code will build, it will not be active unless configured with --enable-bootstrap. If bootstrap is enabled and the new opal_sysinfo framework can successfully determine the cpu model, pass that info to the application as an MCA param to support some work at Sun. Also, have daemons report back the resources they find to guide process mapping in bootstrap operations (i.e., where the daemon starts at node boot as opposed to being launched at application start). Adjust some platform files to enable these capabilities. This commit was SVN r22244.
2009-12-01 02:11:25 +03:00
while (NULL != (item = opal_list_remove_first(&node->resources))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&node->resources);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_node_t,
opal_list_item_t,
orte_node_construct,
orte_node_destruct);
static void orte_proc_construct(orte_proc_t* proc)
{
proc->name = *ORTE_NAME_INVALID;
proc->pid = 0;
Continue implementation of resilient operations by supporting reuse of jobids for restarted procs. Ensure that restarted processes have valid node and local ranks, and that node rank values are passed to direct-launched processes. This commit was SVN r21385.
2009-06-06 05:08:47 +04:00
proc->local_rank = ORTE_LOCAL_RANK_INVALID;
proc->node_rank = ORTE_NODE_RANK_INVALID;
ErrMgr Framework redesign to better support fault tolerance development activities. Explained in more detail in the following RFC: http://www.open-mpi.org/community/lists/devel/2010/03/7589.php This commit was SVN r22872.
2010-03-24 00:28:02 +03:00
proc->last_errmgr_state = ORTE_PROC_STATE_UNDEF;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->state = ORTE_PROC_STATE_UNDEF;
Change the app_idx type to uint32_t to support users who use large numbers of app_contexts. Set it up as a new typedef so we can change it later without as much effort. This commit was SVN r22727.
2010-02-27 20:37:34 +03:00
proc->app_idx = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->slot_list = NULL;
proc->node = NULL;
proc->nodename = NULL;
proc->rml_uri = NULL;
Add the ability to track how many times a process has been restarted, and to communicate that value to a process when it is restarted in case it needs to take action when it is restarted as opposed to being started for the first time. This commit was SVN r22377.
2010-01-07 04:19:44 +03:00
proc->restarts = 0;
Add a sensor framework to ORTE that monitors applications and notifies the errmgr when they exceed specified boundaries. Two modules are included here: 1. file activity - can monitor file size, access and modification times. If these fail to change over a specified number of sampling iterations (rate is an mca param), then the errmgr is notified. 2. memory usage - checks amount of memory used by a process. Limit and sampling rate can be set. This support must be enabled by configuring --enable-sensors. ompi_info and orte-info have been updated to include the new framework. Also includes some initial steps toward restoring the recovery capability. Most notably, the ODLS API has been extended to include a "restart_proc" entry for restarting a local process, and organizes the various ERRMGR framework globals into a single struct as we do in the other ORTE frameworks. Fix an oversight in the ERRMGR framework where a pointer array was constructed, but not initialized. Implementation continues. This commit was SVN r23043.
2010-04-27 02:15:57 +04:00
proc->relocates = 0;
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
proc->ckpt_state = 0;
proc->ckpt_snapshot_ref = NULL;
proc->ckpt_snapshot_loc = NULL;
#endif
}
static void orte_proc_destruct(orte_proc_t* proc)
{
/* do NOT free the nodename field as this is
* simply a pointer to a field in the
* associated node object - the node object
* will free it
*/
if (NULL != proc->slot_list) {
free(proc->slot_list);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->slot_list = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != proc->node) {
OBJ_RELEASE(proc->node);
proc->node = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
if (NULL != proc->rml_uri) {
free(proc->rml_uri);
proc->rml_uri = NULL;
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
Fix the configure logic for --with-ft so that it properly takes a comma separated list. Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those. The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed. This commit was SVN r22824.
2010-03-13 02:57:50 +03:00
#if OPAL_ENABLE_FT_CR == 1
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != proc->ckpt_snapshot_ref) {
free(proc->ckpt_snapshot_ref);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->ckpt_snapshot_ref = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
if (NULL != proc->ckpt_snapshot_loc) {
free(proc->ckpt_snapshot_loc);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
proc->ckpt_snapshot_loc = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
#endif
}
OBJ_CLASS_INSTANCE(orte_proc_t,
opal_list_item_t,
orte_proc_construct,
orte_proc_destruct);
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
static void orte_attr_construct(orte_attr_t *ptr)
{
ptr->name = NULL;
ptr->size = 0;
ptr->bytes = NULL;
}
static void orte_attr_destruct(orte_attr_t *ptr)
{
if (NULL != ptr->name) {
free(ptr->name);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->name = NULL;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
}
if (NULL != ptr->bytes) {
free(ptr->bytes);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->bytes = NULL;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
}
}
OBJ_CLASS_INSTANCE(orte_attr_t,
opal_list_item_t,
orte_attr_construct,
orte_attr_destruct);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
static void orte_nid_construct(orte_nid_t *ptr)
{
ptr->name = NULL;
ptr->daemon = ORTE_VPID_INVALID;
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
OBJ_CONSTRUCT(&ptr->attrs, opal_list_t);
Expand the ESS framework's API to include a new function "query_sys_info" that allows the caller to retrieve key-value pairs of info on the local system capabilities (e.g., cpu type/model). Have each daemon and the HNP "sense" that information and provide it to their local procs to avoid having every proc querying the system directly. This commit was SVN r22870.
2010-03-23 23:47:41 +03:00
OBJ_CONSTRUCT(&ptr->sysinfo, opal_list_t);
Create a new "heartbeat" module in the sensor framework and move the plm_base heartbeat code there. Add new proc and job states for heartbeat_failed. Remove the "heartbeat" cmd line option for orted as this is now done automatically if the --enable-heartbeat configure option is set. This commit was SVN r23102.
2010-05-05 04:48:43 +04:00
#if ORTE_ENABLE_HEARTBEAT
ptr->beat = 0;
ptr->missed = 0;
#endif
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_nid_destruct(orte_nid_t *ptr)
{
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
opal_list_item_t *item;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
if (NULL != ptr->name) {
free(ptr->name);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->name = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose. Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects. This commit was SVN r20219.
2009-01-07 17:58:38 +03:00
while (NULL != (item = opal_list_remove_first(&ptr->attrs))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&ptr->attrs);
Expand the ESS framework's API to include a new function "query_sys_info" that allows the caller to retrieve key-value pairs of info on the local system capabilities (e.g., cpu type/model). Have each daemon and the HNP "sense" that information and provide it to their local procs to avoid having every proc querying the system directly. This commit was SVN r22870.
2010-03-23 23:47:41 +03:00
while (NULL != (item = opal_list_remove_first(&ptr->sysinfo))) {
OBJ_RELEASE(item);
}
OBJ_DESTRUCT(&ptr->sysinfo);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_nid_t,
opal_object_t,
orte_nid_construct,
orte_nid_destruct);
static void orte_pmap_construct(orte_pmap_t *ptr)
{
ptr->node = -1;
Make the orte_pmap_t an object so it can be properly initialized. Adjust the construct function to properly indicate invalid node/local ranks This commit was SVN r20465.
2009-02-06 18:29:33 +03:00
ptr->local_rank = ORTE_LOCAL_RANK_INVALID;
ptr->node_rank = ORTE_NODE_RANK_INVALID;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
OBJ_CLASS_INSTANCE(orte_pmap_t,
opal_object_t,
orte_pmap_construct,
NULL);
static void orte_jmap_construct(orte_jmap_t *ptr)
{
ptr->job = ORTE_JOBID_INVALID;
Complete implementation of regular expression generator and parser - now handles leading zero's and suffix in node names. This commit was SVN r21468.
2009-06-18 08:36:00 +04:00
ptr->num_procs = 0;
Fix a race condition that causes corruption of a buffer in mpirun while trying to process launch_local_proc cmds. Cleanup the pidmap handling by changing from value to pointer arrays. This commit was SVN r20629.
2009-02-25 05:43:22 +03:00
OBJ_CONSTRUCT(&ptr->pmap, opal_pointer_array_t);
opal_pointer_array_init(&ptr->pmap,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
static void orte_jmap_destruct(orte_jmap_t *ptr)
{
Fix a race condition that causes corruption of a buffer in mpirun while trying to process launch_local_proc cmds. Cleanup the pidmap handling by changing from value to pointer arrays. This commit was SVN r20629.
2009-02-25 05:43:22 +03:00
orte_pmap_t **pmaps;
int i;
pmaps = (orte_pmap_t**)ptr->pmap.addr;
for (i=0; i < ptr->pmap.size && NULL != pmaps[i]; i++) {
OBJ_RELEASE(pmaps[i]);
}
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
OBJ_DESTRUCT(&ptr->pmap);
}
OBJ_CLASS_INSTANCE(orte_jmap_t,
opal_object_t,
orte_jmap_construct,
orte_jmap_destruct);
static void orte_job_map_construct(orte_job_map_t* map)
{
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
map->policy = 0;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
map->npernode = 0;
Modify the OMPI paffinity and mapping system to support socket-level mapping and binding. Mostly refactors existing code, with modifications to the odls_default module to support the new capabilities. Adds several new mpirun options: * -bysocket - assign ranks on a node by socket. Effectively load balances the procs assigned to a node across the available sockets. Note that ranks can still be bound to a specific core within the socket, or to the entire socket - the mapping is independent of the binding. * -bind-to-socket - bind each rank to all the cores on the socket to which they are assigned. * -bind-to-core - currently the default behavior (maintained from prior default) * -npersocket N - launch N procs for every socket on a node. Note that this implies we know how many sockets are on a node. Mpirun will determine its local values. These can be overridden by provided values, either via MCA param or in a hostfile Similar features/options are provided at the board level for multi-board nodes. Documentation to follow... This commit was SVN r21791.
2009-08-11 06:51:27 +04:00
map->nperboard = 0;
map->npersocket = 0;
map->cpus_per_rank = 1;
map->stride = 1;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
map->oversubscribe = true; /* default to allowing oversubscribe */
map->display_map = false;
map->cpu_lists = false;
map->num_new_daemons = 0;
map->daemon_vpid_start = ORTE_VPID_INVALID;
map->num_nodes = 0;
map->nodes = OBJ_NEW(opal_pointer_array_t);
opal_pointer_array_init(map->nodes,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE,
ORTE_GLOBAL_ARRAY_MAX_SIZE,
ORTE_GLOBAL_ARRAY_BLOCK_SIZE);
}
static void orte_job_map_destruct(orte_job_map_t* map)
{
orte_std_cntr_t i;
for (i=0; i < map->nodes->size; i++) {
if (NULL != map->nodes->addr[i]) {
OBJ_RELEASE(map->nodes->addr[i]);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
map->nodes->addr[i] = NULL;
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
}
}
OBJ_RELEASE(map->nodes);
}
OBJ_CLASS_INSTANCE(orte_job_map_t,
opal_object_t,
orte_job_map_construct,
orte_job_map_destruct);
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
static void orte_regex_node_construct(orte_regex_node_t *ptr)
{
ptr->prefix = NULL;
Complete implementation of regular expression generator and parser - now handles leading zero's and suffix in node names. This commit was SVN r21468.
2009-06-18 08:36:00 +04:00
OBJ_CONSTRUCT(&ptr->suffix, opal_value_array_t);
opal_value_array_init(&ptr->suffix, sizeof(char));
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
OBJ_CONSTRUCT(&ptr->nodes, opal_value_array_t);
opal_value_array_init(&ptr->nodes, sizeof(int32_t));
OBJ_CONSTRUCT(&ptr->cnt, opal_value_array_t);
opal_value_array_init(&ptr->cnt, sizeof(int32_t));
OBJ_CONSTRUCT(&ptr->starting_vpid, opal_value_array_t);
opal_value_array_init(&ptr->starting_vpid, sizeof(orte_vpid_t));
OBJ_CONSTRUCT(&ptr->ppn, opal_value_array_t);
opal_value_array_init(&ptr->ppn, sizeof(int32_t));
OBJ_CONSTRUCT(&ptr->nrank, opal_value_array_t);
opal_value_array_init(&ptr->nrank, sizeof(orte_node_rank_t));
}
static void orte_regex_node_destruct(orte_regex_node_t *ptr)
{
if (NULL != ptr->prefix) {
free(ptr->prefix);
Make sure to set these values to NULL, so that if we release an object (knowingly or not) twice then we do not fault by referencing unallocated memory. This commit was SVN r21672.
2009-07-14 22:56:49 +04:00
ptr->prefix = NULL;
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
}
Complete implementation of regular expression generator and parser - now handles leading zero's and suffix in node names. This commit was SVN r21468.
2009-06-18 08:36:00 +04:00
OBJ_DESTRUCT(&ptr->suffix);
Add regular expression generator to encode complete nid/pid maps - decoder to come. This commit was SVN r21455.
2009-06-17 06:54:20 +04:00
OBJ_DESTRUCT(&ptr->nodes);
OBJ_DESTRUCT(&ptr->cnt);
OBJ_DESTRUCT(&ptr->starting_vpid);
OBJ_DESTRUCT(&ptr->ppn);
OBJ_DESTRUCT(&ptr->nrank);
}
OBJ_CLASS_INSTANCE(orte_regex_node_t,
opal_list_item_t,
orte_regex_node_construct,
orte_regex_node_destruct);
Follow-on to r19457. Rather than have #ifs in the middle of functions (which neither Ralph nor I liked), don't allow the functions we don't need to be visible. Still not happy about the number of #ifs in the code, but splitting the code further would have been a nightmare and this was a good cutting point. Also protected some variables that were declared but not instanced so that users would be notified at compile time instead of link or run time (in the case of dss constants) that things wouldn't work. This commit was SVN r19471. The following SVN revision numbers were found above: r19457 --> open-mpi/ompi@a15171e46b032a031011ef6695d355f3a8abf496
2008-09-01 21:15:01 +04:00
#endif
Копировать постоянную ссылку