e7ecd56bd2
such, the commit message back to the master SVN repository is fairly
long.
= ORTE Job-Level Output Messages =
Add two new interfaces that should be used for all new code throughout
the ORTE and OMPI layers (we already make the search-and-replace on
the existing ORTE / OMPI layers):
* orte_output(): (and corresponding friends ORTE_OUTPUT,
orte_output_verbose, etc.) This function sends the output directly
to the HNP for processing as part of a job-specific output
channel. It supports all the same outputs as opal_output()
(syslog, file, stdout, stderr), but for stdout/stderr, the output
is sent to the HNP for processing and output. More on this below.
* orte_show_help(): This function is a drop-in-replacement for
opal_show_help(), with two differences in functionality:
1. the rendered text help message output is sent to the HNP for
display (rather than outputting directly into the process' stderr
stream)
1. the HNP detects duplicate help messages and does not display them
(so that you don't see the same error message N times, once from
each of your N MPI processes); instead, it counts "new" instances
of the help message and displays a message every ~5 seconds when
there are new ones ("I got X new copies of the help message...")
opal_show_help and opal_output still exist, but they only output in
the current process. The intent for the new orte_* functions is that
they can apply job-level intelligence to the output. As such, we
recommend that all new ORTE and OMPI code use the new orte_*
functions, not thei opal_* functions.
=== New code ===
For ORTE and OMPI programmers, here's what you need to do differently
in new code:
* Do not include opal/util/show_help.h or opal/util/output.h.
Instead, include orte/util/output.h (this one header file has
declarations for both the orte_output() series of functions and
orte_show_help()).
* Effectively s/opal_output/orte_output/gi throughout your code.
Note that orte_output_open() takes a slightly different argument
list (as a way to pass data to the filtering stream -- see below),
so you if explicitly call opal_output_open(), you'll need to
slightly adapt to the new signature of orte_output_open().
* Literally s/opal_show_help/orte_show_help/. The function signature
is identical.
=== Notes ===
* orte_output'ing to stream 0 will do similar to what
opal_output'ing did, so leaving a hard-coded "0" as the first
argument is safe.
* For systems that do not use ORTE's RML or the HNP, the effect of
orte_output_* and orte_show_help will be identical to their opal
counterparts (the additional information passed to
orte_output_open() will be lost!). Indeed, the orte_* functions
simply become trivial wrappers to their opal_* counterparts. Note
that we have not tested this; the code is simple but it is quite
possible that we mucked something up.
= Filter Framework =
Messages sent view the new orte_* functions described above and
messages output via the IOF on the HNP will now optionally be passed
through a new "filter" framework before being output to
stdout/stderr. The "filter" OPAL MCA framework is intended to allow
preprocessing to messages before they are sent to their final
destinations. The first component that was written in the filter
framework was to create an XML stream, segregating all the messages
into different XML tags, etc. This will allow 3rd party tools to read
the stdout/stderr from the HNP and be able to know exactly what each
text message is (e.g., a help message, another OMPI infrastructure
message, stdout from the user process, stderr from the user process,
etc.).
Filtering is not active by default. Filter components must be
specifically requested, such as:
{{{
$ mpirun --mca filter xml ...
}}}
There can only be one filter component active.
= New MCA Parameters =
The new functionality described above introduces two new MCA
parameters:
* '''orte_base_help_aggregate''': Defaults to 1 (true), meaning that
help messages will be aggregated, as described above. If set to 0,
all help messages will be displayed, even if they are duplicates
(i.e., the original behavior).
* '''orte_base_show_output_recursions''': An MCA parameter to help
debug one of the known issues, described below. It is likely that
this MCA parameter will disappear before v1.3 final.
= Known Issues =
* The XML filter component is not complete. The current output from
this component is preliminary and not real XML. A bit more work
needs to be done to configure.m4 search for an appropriate XML
library/link it in/use it at run time.
* There are possible recursion loops in the orte_output() and
orte_show_help() functions -- e.g., if RML send calls orte_output()
or orte_show_help(). We have some ideas how to fix these, but
figured that it was ok to commit before feature freeze with known
issues. The code currently contains sub-optimal workarounds so
that this will not be a problem, but it would be good to actually
solve the problem rather than have hackish workarounds before v1.3 final.
This commit was SVN r18434.
258 строки
8.3 KiB
C
258 строки
8.3 KiB
C
/*
|
|
* Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana
|
|
* University Research and Technology
|
|
* Corporation. All rights reserved.
|
|
* Copyright (c) 2004-2005 The University of Tennessee and The University
|
|
* of Tennessee Research Foundation. All rights
|
|
* reserved.
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
* University of Stuttgart. All rights reserved.
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
* All rights reserved.
|
|
* $COPYRIGHT$
|
|
*
|
|
* Additional copyrights may follow
|
|
*
|
|
* $HEADER$
|
|
*/
|
|
/**
|
|
* @file
|
|
*
|
|
* Snapshot Coordination (SNAPC) Interface
|
|
*
|
|
* Terminology:
|
|
* ------------
|
|
* Global Snapshot Coordinator:
|
|
* - HNP(s) coordination function.
|
|
* Local Snapshot Coordinator
|
|
* - VHNP(s) [e.g., orted] coordination function
|
|
* Application Snapshot Coordinator
|
|
* - Application level coordinaton function
|
|
* Local Snapshot
|
|
* - Snapshot generated by a single process in the parallel job
|
|
* Local Snapshot Reference
|
|
* - A generic reference to the physical Local Snapshot
|
|
* Global Snapshot
|
|
* - Snapshot generated for the entire parallel job
|
|
* Global Snapshot Reference
|
|
* - A generic reference to the physical Global Snapshot
|
|
*
|
|
* General Description:
|
|
* ---------------------
|
|
* This framework is tasked with:
|
|
* - Initiating the checkpoint in the system
|
|
* - Physically moving the local snapshot files to a location
|
|
* Initially this location, is the node on which the Head Node Process (HNP)
|
|
* is running, but later this will be a replicated checkpoint server or
|
|
* the like.
|
|
* - Generating a 'global snapshot handle' that the user can use to restart
|
|
* the parallel job.
|
|
*
|
|
* Each component will have 3 teirs of behavior that must behave in concert:
|
|
* - Global Snapshot Coordinator
|
|
* This is the HNPs tasks. Mostly distributing the notification of the
|
|
* checkpoint, and then compiling the physical and virtual nature of the
|
|
* global snapshot handle.
|
|
* - Local Snapshot Coordinator
|
|
* This is the VHNPs (or orted, if available) tasks. This will involve
|
|
* working with the Global Snapshot Coordinator to route the physical
|
|
* and virtual 'local snapshot's from the application to the desired
|
|
* location. This process must also notify the Global Snapshot Coordinator
|
|
* when it's set of processes have completed the checkpoint.
|
|
* - Application Snapshot Coordinator
|
|
* This is the application level coordinator. This is very light, just
|
|
* a subscription to be triggered when it needs to checkpoint, and then,
|
|
* once finished with the checkpoint, notify the Local Snapshot Coordinator
|
|
* that it is complete.
|
|
* If there is no orted (so no bootproxy), then the application assumes the
|
|
* responsibility of the Local Snapshot Coordinator as well.
|
|
*
|
|
*/
|
|
|
|
#ifndef MCA_SNAPC_H
|
|
#define MCA_SNAPC_H
|
|
|
|
#include "orte_config.h"
|
|
#include "orte/constants.h"
|
|
#include "orte/types.h"
|
|
|
|
#include "orte/util/output.h"
|
|
#include "opal/mca/mca.h"
|
|
#include "opal/mca/base/base.h"
|
|
#include "opal/mca/crs/crs.h"
|
|
#include "opal/mca/crs/base/base.h"
|
|
|
|
#include "opal/class/opal_object.h"
|
|
|
|
BEGIN_C_DECLS
|
|
|
|
/**
|
|
* States that a process can be in while checkpointing
|
|
*/
|
|
/* Doing no checkpoint -- Quiet state */
|
|
#define ORTE_SNAPC_CKPT_STATE_NONE 0
|
|
/* There has been a request for a checkpoint from one of the applications */
|
|
#define ORTE_SNAPC_CKPT_STATE_REQUEST 1
|
|
/* There is a Pending checkpoint for this process */
|
|
#define ORTE_SNAPC_CKPT_STATE_PENDING 2
|
|
/* There is a Pending checkpoint for this process, terminate the process after checkpoint */
|
|
#define ORTE_SNAPC_CKPT_STATE_PENDING_TERM 3
|
|
/* Running the checkpoint */
|
|
#define ORTE_SNAPC_CKPT_STATE_RUNNING 4
|
|
/* Finished the checkpoint */
|
|
#define ORTE_SNAPC_CKPT_STATE_FILE_XFER 5
|
|
/* Finished the checkpoint */
|
|
#define ORTE_SNAPC_CKPT_STATE_FINISHED 6
|
|
/* Unable to checkpoint this job */
|
|
#define ORTE_SNAPC_CKPT_STATE_NO_CKPT 7
|
|
/* Reached an error */
|
|
#define ORTE_SNAPC_CKPT_STATE_ERROR 8
|
|
|
|
/**
|
|
* Definition of a orte local snapshot.
|
|
* Similar to the opal_crs_base_snapshot_t except that it
|
|
* contains process contact information.
|
|
*/
|
|
struct orte_snapc_base_snapshot_1_0_0_t {
|
|
opal_crs_base_snapshot_t crs_snapshot_super;
|
|
/** ORTE Process name */
|
|
orte_process_name_t process_name;
|
|
/** PID of the application process that generated this snapshot */
|
|
pid_t process_pid;
|
|
/** State of the checkpoint */
|
|
size_t state;
|
|
/** Terminate this process after a checkpoint */
|
|
bool term;
|
|
};
|
|
typedef struct orte_snapc_base_snapshot_1_0_0_t orte_snapc_base_snapshot_1_0_0_t;
|
|
typedef struct orte_snapc_base_snapshot_1_0_0_t orte_snapc_base_snapshot_t;
|
|
|
|
ORTE_DECLSPEC OBJ_CLASS_DECLARATION(orte_snapc_base_snapshot_t);
|
|
|
|
/**
|
|
* Definition of the global snapshot.
|
|
* Each component is assumed to have extened this definition
|
|
* in the same way they extern the orte_snapc_base_compoinent_t below.
|
|
*/
|
|
struct orte_snapc_base_global_snapshot_1_0_0_t {
|
|
/** This is an object, so must have super */
|
|
opal_list_item_t super;
|
|
|
|
/** A list of orte_snapc_base_snapshot_ts */
|
|
opal_list_t snapshots;
|
|
|
|
/* ORTE SnapC Component used to generate the global snapshot */
|
|
char * component_name;
|
|
|
|
/** Unique name of the global snapshot */
|
|
char * reference_name;
|
|
|
|
/** Location of the global snapshot Absolute path */
|
|
char * local_location;
|
|
|
|
/** Sequence Number */
|
|
int seq_num;
|
|
|
|
/** Beginning timestamp */
|
|
char * start_time;
|
|
|
|
/** Ending timestamp */
|
|
char * end_time;
|
|
};
|
|
typedef struct orte_snapc_base_global_snapshot_1_0_0_t orte_snapc_base_global_snapshot_1_0_0_t;
|
|
typedef struct orte_snapc_base_global_snapshot_1_0_0_t orte_snapc_base_global_snapshot_t;
|
|
|
|
ORTE_DECLSPEC OBJ_CLASS_DECLARATION(orte_snapc_base_global_snapshot_t);
|
|
|
|
/**
|
|
* Module initialization function.
|
|
* Returns ORTE_SUCCESS
|
|
*/
|
|
typedef int (*orte_snapc_base_module_init_fn_t)
|
|
(bool seed, bool app);
|
|
|
|
/**
|
|
* Module finalization function.
|
|
* Returns ORTE_SUCCESS
|
|
*/
|
|
typedef int (*orte_snapc_base_module_finalize_fn_t)
|
|
(void);
|
|
|
|
/**
|
|
* Setup the necessary structures for this job
|
|
* Returns ORTE_SUCCESS
|
|
*/
|
|
typedef int (*orte_snapc_base_setup_job_fn_t)
|
|
(orte_jobid_t jobid);
|
|
|
|
/**
|
|
* Setup the necessary structures for this job
|
|
* Returns ORTE_SUCCESS
|
|
*/
|
|
typedef int (*orte_snapc_base_release_job_fn_t)
|
|
(orte_jobid_t jobid);
|
|
|
|
|
|
/**
|
|
* Handle fault tolerance updates
|
|
*
|
|
* @param[in] state Fault tolerance state update
|
|
*
|
|
* @retval ORTE_SUCCESS The operation completed successfully
|
|
* @retval ORTE_ERROR An unspecifed error occurred
|
|
*/
|
|
typedef int (*orte_snapc_base_ft_event_fn_t)(int state);
|
|
|
|
/**
|
|
* Structure for SNAPC v1.0.0 components.
|
|
*/
|
|
struct orte_snapc_base_component_1_0_0_t {
|
|
/** MCA base component */
|
|
mca_base_component_t base_version;
|
|
/** MCA base data */
|
|
mca_base_component_data_1_0_0_t base_data;
|
|
|
|
/** Verbosity Level */
|
|
int verbose;
|
|
/** Output Handle for orte_output */
|
|
int output_handle;
|
|
/** Default Priority */
|
|
int priority;
|
|
};
|
|
typedef struct orte_snapc_base_component_1_0_0_t orte_snapc_base_component_1_0_0_t;
|
|
typedef struct orte_snapc_base_component_1_0_0_t orte_snapc_base_component_t;
|
|
|
|
/**
|
|
* Structure for SNAPC v1.0.0 modules
|
|
*/
|
|
struct orte_snapc_base_module_1_0_0_t {
|
|
/** Initialization Function */
|
|
orte_snapc_base_module_init_fn_t snapc_init;
|
|
/** Finalization Function */
|
|
orte_snapc_base_module_finalize_fn_t snapc_finalize;
|
|
/** Setup structures for a job */
|
|
orte_snapc_base_setup_job_fn_t setup_job;
|
|
/** Release job */
|
|
orte_snapc_base_release_job_fn_t release_job;
|
|
/** Handle any FT Notifications */
|
|
orte_snapc_base_ft_event_fn_t ft_event;
|
|
};
|
|
typedef struct orte_snapc_base_module_1_0_0_t orte_snapc_base_module_1_0_0_t;
|
|
typedef struct orte_snapc_base_module_1_0_0_t orte_snapc_base_module_t;
|
|
|
|
ORTE_DECLSPEC extern orte_snapc_base_module_t orte_snapc;
|
|
|
|
/**
|
|
* Macro for use in components that are of type SNAPC v1.0.0
|
|
*/
|
|
#define ORTE_SNAPC_BASE_VERSION_1_0_0 \
|
|
/* SNAPC v1.0 is chained to MCA v1.0 */ \
|
|
MCA_BASE_VERSION_1_0_0, \
|
|
/* SNAPC v1.0 */ \
|
|
"snapc", 1, 0, 0
|
|
|
|
END_C_DECLS
|
|
|
|
#endif /* ORTE_SNAPC_H */
|
|
|