2007-03-17 02:11:45 +03:00
|
|
|
.\"
|
A number of C/R enhancements per RFC below:
http://www.open-mpi.org/community/lists/devel/2010/07/8240.php
Documentation:
http://osl.iu.edu/research/ft/
Major Changes:
--------------
* Added C/R-enabled Debugging support.
Enabled with the --enable-crdebug flag. See the following website for more information:
http://osl.iu.edu/research/ft/crdebug/
* Added Stable Storage (SStore) framework for checkpoint storage
* 'central' component does a direct to central storage save
* 'stage' component stages checkpoints to central storage while the application continues execution.
* 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress)
* 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching)
* Added Compression (compress) framework to support
* Add two new ErrMgr recovery policies
* {{{crmig}}} C/R Process Migration
* {{{autor}}} C/R Automatic Recovery
* Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component
* Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option)
* {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342)
* {{{OMPI_CR_Restart}}}
* {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules)
* {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192)
* {{{OMPI_CR_Quiesce_start}}}
* {{{OMPI_CR_Quiesce_checkpoint}}}
* {{{OMPI_CR_Quiesce_end}}}
* {{{OMPI_CR_self_register_checkpoint_callback}}}
* {{{OMPI_CR_self_register_restart_callback}}}
* {{{OMPI_CR_self_register_continue_callback}}}
* The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future.
* Add a progress meter to:
* FileM rsh (filem_rsh_process_meter)
* SnapC full (snapc_full_progress_meter)
* SStore stage (sstore_stage_progress_meter)
* Added 2 new command line options to ompi-restart
* --showme : Display the full command line that would have been exec'ed.
* --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413)
* Deprecated some MCA params:
* crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir
* snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir
* snapc_base_global_shared deprecated, use sstore_stage_global_is_shared
* snapc_base_store_in_place deprecated, replaced with different components of SStore
* snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref
* snapc_base_establish_global_snapshot_dir deprecated, never well supported
* snapc_full_skip_filem deprecated, use sstore_stage_skip_filem
Minor Changes:
--------------
* Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing.
* Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components
* Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it.
* Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}}
* Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set.
* opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality.
* Cleanup the CRS framework and components to work with the SStore framework.
* Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably).
* Add 'quiesce' hook to CRCP for a future enhancement.
* We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}.
* Add optional application level INC callbacks (registered through the CR MPI Ext interface).
* Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive.
* {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked.
* {{{opal-restart}}} also support local decompression before restarting
* {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata
* {{{orte-restart}}} now uses the SStore framework to work with the metadata
* Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality.
* Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}.
* Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped.
* Make sure to decrement the number of 'num_local_procs' in the orted when one goes away.
* odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options.
* Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities.
* Improve the checks for 'already checkpointing' error path.
* A a recovery output timer, to show how long it takes to restart a job
* Do a better job of cleaning up the old session directory on restart.
* Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment)
* Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize.
This commit was SVN r23587.
The following Trac tickets were found above:
Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924
Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097
Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161
Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192
Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208
Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342
Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
2010-08-11 00:51:11 +04:00
|
|
|
.\" Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
2007-03-17 02:11:45 +03:00
|
|
|
.\" University Research and Technology
|
|
|
|
|
.\" Corporation. All rights reserved.
|
2009-04-01 18:40:27 +04:00
|
|
|
.\" Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved.
|
2007-03-17 02:11:45 +03:00
|
|
|
.\"
|
|
|
|
|
.\" Man page for OPAL's CRS Functionality
|
|
|
|
|
.\"
|
|
|
|
|
.\" .TH name section center-footer left-footer center-header
|
George did the work and deserves all the credit for it. Ralph did the merge, and deserves whatever blame results from errors in it :-)
WHAT: Open our low-level communication infrastructure by moving all necessary components (btl/rcache/allocator/mpool) down in OPAL
All the components required for inter-process communications are currently deeply integrated in the OMPI layer. Several groups/institutions have express interest in having a more generic communication infrastructure, without all the OMPI layer dependencies. This communication layer should be made available at a different software level, available to all layers in the Open MPI software stack. As an example, our ORTE layer could replace the current OOB and instead use the BTL directly, gaining access to more reactive network interfaces than TCP. Similarly, external software libraries could take advantage of our highly optimized AM (active message) communication layer for their own purpose. UTK with support from Sandia, developped a version of Open MPI where the entire communication infrastucture has been moved down to OPAL (btl/rcache/allocator/mpool). Most of the moved components have been updated to match the new schema, with few exceptions (mainly BTLs where I have no way of compiling/testing them). Thus, the completion of this RFC is tied to being able to completing this move for all BTLs. For this we need help from the rest of the Open MPI community, especially those supporting some of the BTLs. A non-exhaustive list of BTLs that qualify here is: mx, portals4, scif, udapl, ugni, usnic.
This commit was SVN r32317.
2014-07-26 04:47:28 +04:00
|
|
|
.TH OPAL_CRS 7 "#OPAL_DATE#" "#PACKAGE_VERSION#" "#PACKAGE_NAME#"
|
2008-08-02 01:14:37 +04:00
|
|
|
|
2007-03-17 02:11:45 +03:00
|
|
|
.\" **************************
|
|
|
|
|
.\" Name Section
|
|
|
|
|
.\" **************************
|
|
|
|
|
.SH NAME
|
|
|
|
|
.
|
2010-07-20 18:07:18 +04:00
|
|
|
OPAL_CRS \- Open PAL MCA Checkpoint/Restart Service (CRS): Overview of Open PAL's
|
|
|
|
|
CRS framework, and selected modules. #PACKAGE_NAME# #PACKAGE_VERSION#.
|
2007-03-17 02:11:45 +03:00
|
|
|
.
|
|
|
|
|
.\" **************************
|
|
|
|
|
.\" Description Section
|
|
|
|
|
.\" **************************
|
|
|
|
|
.SH DESCRIPTION
|
|
|
|
|
.
|
|
|
|
|
.PP
|
|
|
|
|
Open PAL can involuntarily checkpoint and restart sequential programs.
|
|
|
|
|
Doing so requires that Open PAL was compiled with thread support and
|
|
|
|
|
that the back-end checkpointing systems are available at run-time.
|
|
|
|
|
.
|
|
|
|
|
.SS Phases of Checkpoint / Restart
|
|
|
|
|
.PP
|
|
|
|
|
Open PAL defines three phases for checkpoint / restart support in a
|
|
|
|
|
procress:
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
Checkpoint
|
|
|
|
|
When the checkpoint request arrives, the procress is notified of the
|
|
|
|
|
request before the checkpoint is taken.
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
Continue
|
|
|
|
|
After a checkpoint has successfully completed, the same process as the
|
|
|
|
|
checkpoint is notified of its successful continuation of execution.
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
Restart
|
|
|
|
|
After a checkpoint has successfully completed, a new / restarted
|
|
|
|
|
process is notified of its successful restart.
|
|
|
|
|
.
|
|
|
|
|
.PP
|
|
|
|
|
The Continue and Restart phases are identical except for the process
|
|
|
|
|
in which they are invoked. The Continue phase is invoked in the same process
|
|
|
|
|
as the Checkpoint phase was invoked. The Restart phase is only invoked in newly
|
|
|
|
|
restarted processes.
|
|
|
|
|
.
|
|
|
|
|
.\" **************************
|
|
|
|
|
.\" General Process Requirements Section
|
|
|
|
|
.\" **************************
|
|
|
|
|
.SH GENERAL PROCESS REQUIREMENTS
|
|
|
|
|
.PP
|
|
|
|
|
In order for a process to use the Open PAL CRS components it must adhear to a
|
|
|
|
|
few programmatic requirements.
|
|
|
|
|
.PP
|
|
|
|
|
First, the program must call \fIOPAL_INIT\fR early in its execution. This
|
|
|
|
|
should only be called once, and it is not possible to checkpoint the process
|
|
|
|
|
without it first having called this function.
|
|
|
|
|
.PP
|
|
|
|
|
The program must call \fIOPAL_FINALIZE\fR before termination. This does a
|
|
|
|
|
significant amount of cleanup. If it is not called, then it is very likely that
|
|
|
|
|
remnants are left in the filesystem.
|
|
|
|
|
.PP
|
|
|
|
|
To checkpoint and restart a process you must use the Open PAL tools to do
|
|
|
|
|
so. Using the backend checkpointer's checkpoint and restart tools will lead
|
|
|
|
|
to undefined behavior.
|
|
|
|
|
To checkpoint a process use \fIopal_checkpoint\fR (opal_checkpoint(1)).
|
|
|
|
|
To restart a process use \fIopal_restart\fR (opal_restart(1)).
|
|
|
|
|
.
|
|
|
|
|
.\" **********************************
|
|
|
|
|
.\" Available Components Section
|
|
|
|
|
.\" **********************************
|
|
|
|
|
.SH AVAILABLE COMPONENTS
|
|
|
|
|
.PP
|
|
|
|
|
Open PAL ships with two CRS components: \fIself\fR and \fIblcr\fR.
|
|
|
|
|
.
|
|
|
|
|
.PP
|
|
|
|
|
The following MCA parameters apply to all components:
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
crs_base_verbose
|
|
|
|
|
Set the verbosity level for all components. Default is 0, or silent except on error.
|
|
|
|
|
.
|
|
|
|
|
.\" Self Component
|
|
|
|
|
.\" ******************
|
|
|
|
|
.SS self CRS Component
|
|
|
|
|
.PP
|
|
|
|
|
The \fIself\fR component invokes user-defined functions to save and restore
|
|
|
|
|
checkpoints. It is simply a mechanism for user-defined functions to be invoked
|
|
|
|
|
at Open PAL's Checkpoint, Continue, and Restart phases. Hence, the only data
|
|
|
|
|
that is saved during the checkpoint is what is written in the user's checkpoint
|
|
|
|
|
function. No libary state is saved at all.
|
|
|
|
|
.
|
|
|
|
|
.PP
|
|
|
|
|
As such, the model for the \fIself\fR component is slightly differnt than for
|
|
|
|
|
other components. Specifically, the Restart function is not invoked in the same
|
|
|
|
|
process image of the process that was checkpointed. The Restart phase is
|
|
|
|
|
invoked during \fBOPAL_INIT\fR of the new instance of the applicaiton (i.e., it
|
|
|
|
|
starts over from main()).
|
|
|
|
|
.
|
|
|
|
|
.PP
|
|
|
|
|
The \fIself\fR component has the following MCA parameters:
|
|
|
|
|
.TP 4
|
|
|
|
|
crs_self_prefix
|
|
|
|
|
Speficy a string prefix for the name of the checkpoint, continue, and restart
|
|
|
|
|
functions that Open PAL will invoke during the respective stages. That is,
|
|
|
|
|
by specifying "-mca crs_self_prefix foo" means that Open PAL expects to find
|
|
|
|
|
three functions at run-time:
|
|
|
|
|
|
|
|
|
|
int foo_checkpoint()
|
|
|
|
|
|
|
|
|
|
int foo_continue()
|
|
|
|
|
|
|
|
|
|
int foo_restart()
|
|
|
|
|
|
|
|
|
|
By default, the prefix is set to "opal_crs_self_user".
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
crs_self_priority
|
|
|
|
|
Set the \fIself\fR components default priority
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
crs_self_verbose
|
|
|
|
|
Set the verbosity level. Default is 0, or silent except on error.
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
crs_self_do_restart
|
|
|
|
|
This is mostly internally used. A general user should never need to set this
|
|
|
|
|
value. This is set to non-0 when a the new process should invoke the restart
|
|
|
|
|
callback in \fIOPAL_INIT\fR. Default is 0, or normal execution.
|
|
|
|
|
.
|
|
|
|
|
.\" BLCR Component
|
|
|
|
|
.\" ******************
|
|
|
|
|
.SS blcr CRS Component
|
|
|
|
|
.PP
|
|
|
|
|
The Berkeley Lab Checkpoint/Restart (BLCR) single-process checkpoint is a
|
|
|
|
|
software system developed at Lawrence Berkeley National Laboratory. See the
|
|
|
|
|
project website for more details:
|
|
|
|
|
|
|
|
|
|
\fI http://ftg.lbl.gov/CheckpointRestart/CheckpointRestart.shtml \fR
|
|
|
|
|
.
|
|
|
|
|
.PP
|
|
|
|
|
The \fIblcr\fR component has the following MCA parameters:
|
|
|
|
|
.TP 4
|
|
|
|
|
crs_blcr_priority
|
|
|
|
|
Set the \fIblcr\fR components default priority.
|
|
|
|
|
.
|
|
|
|
|
.TP 4
|
|
|
|
|
crs_blcr_verbose
|
|
|
|
|
Set the verbosity level. Default is 0, or silent except on error.
|
|
|
|
|
.
|
|
|
|
|
.\" Special 'none' option
|
|
|
|
|
.\" ************************
|
|
|
|
|
.SS none CRS Component
|
|
|
|
|
.PP
|
|
|
|
|
The \fInone\fP component simply selects no CRS component. All of the CRS
|
|
|
|
|
function calls return immediately with OPAL_SUCCESS.
|
|
|
|
|
.
|
|
|
|
|
.PP
|
|
|
|
|
This component is the last component to be selected by default. This means that if
|
|
|
|
|
another component is available, and the \fInone\fP component was not explicity
|
|
|
|
|
requested then OPAL will attempt to activate all of the available components
|
|
|
|
|
before falling back to this component.
|
|
|
|
|
.
|
|
|
|
|
.\" **************************
|
|
|
|
|
.\" See Also Section
|
|
|
|
|
.\" **************************
|
|
|
|
|
.
|
|
|
|
|
.SH SEE ALSO
|
|
|
|
|
opal_checkpoint(1), opal_restart(1)
|
|
|
|
|
.\", orte_crs(7), ompi_crs(7)
|