1
1

A number of C/R enhancements per RFC below:

http://www.open-mpi.org/community/lists/devel/2010/07/8240.php

Documentation:
  http://osl.iu.edu/research/ft/

Major Changes: 
-------------- 
 * Added C/R-enabled Debugging support. 
   Enabled with the --enable-crdebug flag. See the following website for more information: 
   http://osl.iu.edu/research/ft/crdebug/ 
 * Added Stable Storage (SStore) framework for checkpoint storage 
   * 'central' component does a direct to central storage save 
   * 'stage' component stages checkpoints to central storage while the application continues execution. 
     * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) 
     * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) 
 * Added Compression (compress) framework to support 
 * Add two new ErrMgr recovery policies 
   * {{{crmig}}} C/R Process Migration 
   * {{{autor}}} C/R Automatic Recovery 
 * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component 
 * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) 
   * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) 
   * {{{OMPI_CR_Restart}}} 
   * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) 
   * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) 
   * {{{OMPI_CR_Quiesce_start}}} 
   * {{{OMPI_CR_Quiesce_checkpoint}}} 
   * {{{OMPI_CR_Quiesce_end}}} 
   * {{{OMPI_CR_self_register_checkpoint_callback}}} 
   * {{{OMPI_CR_self_register_restart_callback}}} 
   * {{{OMPI_CR_self_register_continue_callback}}} 
 * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. 
 * Add a progress meter to: 
   * FileM rsh (filem_rsh_process_meter) 
   * SnapC full (snapc_full_progress_meter) 
   * SStore stage (sstore_stage_progress_meter) 
 * Added 2 new command line options to ompi-restart 
   * --showme : Display the full command line that would have been exec'ed. 
   * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) 
 * Deprecated some MCA params: 
   * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir 
   * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir 
   * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared 
   * snapc_base_store_in_place deprecated, replaced with different components of SStore 
   * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref 
   * snapc_base_establish_global_snapshot_dir deprecated, never well supported 
   * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem 

Minor Changes: 
-------------- 
 * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. 
 * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components 
 * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. 
 * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} 
 * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. 
 * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. 
 * Cleanup the CRS framework and components to work with the SStore framework. 
 * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). 
 * Add 'quiesce' hook to CRCP for a future enhancement. 
 * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. 
 * Add optional application level INC callbacks (registered through the CR MPI Ext interface). 
 * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. 
 * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. 
 * {{{opal-restart}}} also support local decompression before restarting 
 * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata 
 * {{{orte-restart}}} now uses the SStore framework to work with the metadata 
 * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. 
 * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. 
 * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. 
 * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. 
 * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. 
 * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. 
 * Improve the checks for 'already checkpointing' error path. 
 * A a recovery output timer, to show how long it takes to restart a job 
 * Do a better job of cleaning up the old session directory on restart. 
 * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) 
 * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. 

This commit was SVN r23587.

The following Trac tickets were found above:
  Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924
  Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097
  Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161
  Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192
  Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208
  Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342
  Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
This commit is contained in:
Josh Hursey 2010-08-10 20:51:11 +00:00
parent 9fff01704f
commit e12ca48cd9
182 changed files with 25548 additions and 3640 deletions

View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
# Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
# University Research and Technology
# Corporation. All rights reserved.
# Copyright (c) 2004-2005 The University of Tennessee and The University
@ -22,7 +22,9 @@ amca_paramdir = $(AMCA_PARAM_SETS_DIR)
dist_amca_param_DATA = amca-param-sets/example.conf
if WANT_FT
dist_amca_param_DATA += amca-param-sets/ft-enable-cr
dist_amca_param_DATA += \
amca-param-sets/ft-enable-cr \
amca-param-sets/ft-enable-cr-recovery
endif
EXTRA_DIST = \

View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2008-2009 The Trustees of Indiana University and Indiana
# Copyright (c) 2008-2010 The Trustees of Indiana University and Indiana
# University Research and Technology
# Corporation. All rights reserved.
#
@ -37,7 +37,6 @@ opal_cr_use_thread=1
#
rml_wrapper=ftrm
snapc=full
#filem=rsh
#
# OMPI Parameters

View File

@ -0,0 +1,82 @@
#
# Copyright (c) 2009-2010 The Trustees of Indiana University and Indiana
# University Research and Technology
# Corporation. All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
# An Aggregate MCA Parameter Set to enable checkpoint/restart capabilities
# for a job.
#
# Usage:
# shell$ mpirun -am ft-enable-cr ./app
#
#
# OPAL Parameters
# - Turn off OPAL only checkpointing
# - Select only checkpoint ready components
# - Enable Additional FT infrastructure
# - Auto-select OPAL CRS component
# - If available, use the FT Thread (Default)
#
opal_cr_allow_opal_only=0
mca_base_component_distill_checkpoint_ready=1
ft_cr_enabled=1
crs=
opal_cr_use_thread=1
#
# ORTE Parameters
# - Wrap the RML
# - Use the 'full' Snapshot Coordinator
# - Use the 'cm' routed component. It is the only one that is currently able to
# handle process and daemon loss.
#
rml_wrapper=ftrm
snapc=full
routed=cm
#
# OMPI Parameters
# - Wrap the PML
# - Use a Bookmark Exchange Fully Coordinated Checkpoint/Restart Coordination Protocol
#
pml_wrapper=crcpw
crcp=bkmrk
#
# Temporary fix to force the event engine to use poll to behave well with BLCR
#
opal_event_include=poll
#
# We currently only support the following options to the OpenIB BTL
# Future development will attempt to eliminate many of these restrictions
#
btl_openib_want_fork_support=1
btl_openib_use_async_event_thread=0
btl_openib_use_eager_rdma=0
btl_openib_cpc_include=oob
# Enable SIGTSTP/SIGCONT capability
# killall -TSTP mpirun
# killall -CONT mpirun
orte_forward_job_control=1
#
# Use the C/R Error Management and Recovery Service
#
orte_enable_recovery=1
orte_max_global_restarts=10
errmgr_crmig_enable=1
errmgr_autor_enable=1
#
# Additional constraints to be lifted in the future
#
plm=rsh
rmaps=resilient

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* Copyright (c) 2004-2007 The University of Tennessee and The University
@ -54,7 +54,7 @@ int mca_bml_r2_ft_event(int state)
first_continue_pass = !first_continue_pass;
/* Since nothing in Checkpoint, we are fine here (unless required by BTL) */
if( ompi_cr_continue_like_restart && !first_continue_pass) {
if( orte_cr_continue_like_restart && !first_continue_pass) {
procs = ompi_proc_all(&num_procs);
if(NULL == procs) {
return OMPI_ERR_OUT_OF_RESOURCE;
@ -136,7 +136,7 @@ int mca_bml_r2_ft_event(int state)
}
else if(OPAL_CRS_CONTINUE == state) {
/* Matches OPAL_CRS_RESTART_PRE */
if( ompi_cr_continue_like_restart && first_continue_pass) {
if( orte_cr_continue_like_restart && first_continue_pass) {
if( OMPI_SUCCESS != (ret = mca_bml_r2_finalize()) ) {
opal_output(0, "bml:r2: ft_event(Restart): Failed to finalize BML framework\n");
return ret;
@ -147,7 +147,7 @@ int mca_bml_r2_ft_event(int state)
}
}
/* Matches OPAL_CRS_RESTART */
else if( ompi_cr_continue_like_restart && !first_continue_pass ) {
else if( orte_cr_continue_like_restart && !first_continue_pass ) {
/*
* Barrier to make all processes have been successfully restarted before
* we try to remove some restart only files.
@ -157,10 +157,6 @@ int mca_bml_r2_ft_event(int state)
return ret;
}
opal_output_verbose(10, ompi_cr_output,
"bml:r2: ft_event(Restart): Cleanup restart files\n");
opal_crs_base_cleanup_flush();
/*
* Re-open the BTL framework to get the full list of components.
*/
@ -234,10 +230,6 @@ int mca_bml_r2_ft_event(int state)
return ret;
}
opal_output_verbose(10, ompi_cr_output,
"bml:r2: ft_event(Restart): Cleanup restart files\n");
opal_crs_base_cleanup_flush();
/*
* Re-open the BTL framework to get the full list of components.
* - but first clear the MCA value that was there

View File

@ -641,7 +641,7 @@ int mca_btl_mx_ft_event(int state) {
* kernel: blcr: thaw_threads returned error, aborting. -1
* JJH: It may be possible to, instead of restarting the entire driver, just reconnect endpoints
*/
ompi_cr_continue_like_restart = true;
orte_cr_continue_like_restart = true;
for( i = 0; i < mca_btl_mx_component.mx_num_btls; i++ ) {
mx_btl = mca_btl_mx_component.mx_btls[i];

View File

@ -1735,7 +1735,7 @@ int mca_btl_openib_ft_event(int state) {
if(OPAL_CRS_CHECKPOINT == state) {
/* Continue must reconstruct the routes (including modex), since we
* have to tear down the devices completely. */
ompi_cr_continue_like_restart = true;
orte_cr_continue_like_restart = true;
/*
* To keep the node from crashing we need to call ibv_close_device

View File

@ -52,6 +52,7 @@
#if OPAL_ENABLE_FT_CR == 1
#include "opal/mca/crs/base/base.h"
#include "opal/util/basename.h"
#include "orte/mca/sstore/sstore.h"
#include "ompi/runtime/ompi_cr.h"
#endif
@ -1099,8 +1100,6 @@ int mca_btl_sm_ft_event(int state) {
}
#else
int mca_btl_sm_ft_event(int state) {
char * tmp_dir = NULL;
/* Notify mpool */
if( NULL != mca_btl_sm_component.sm_mpool &&
NULL != mca_btl_sm_component.sm_mpool->mpool_ft_event) {
@ -1114,17 +1113,14 @@ int mca_btl_sm_ft_event(int state) {
* for these old file handles. The restart procedure will make sure
* these files get cleaned up appropriately.
*/
opal_crs_base_metadata_write_token(NULL, CRS_METADATA_TOUCH, mca_btl_sm_component.sm_seg->module_seg_path);
/* Record the job session directory */
opal_crs_base_metadata_write_token(NULL, CRS_METADATA_MKDIR, orte_process_info.job_session_dir);
orte_sstore.set_attr(orte_sstore_handle_current,
SSTORE_METADATA_LOCAL_TOUCH,
mca_btl_sm_component.sm_seg->module_seg_path);
}
}
else if(OPAL_CRS_CONTINUE == state) {
if( ompi_cr_continue_like_restart ) {
if( orte_cr_continue_like_restart ) {
if( NULL != mca_btl_sm_component.sm_seg ) {
/* Do not Add session directory on continue */
/* Add shared memory file */
opal_crs_base_cleanup_append(mca_btl_sm_component.sm_seg->module_seg_path, false);
}
@ -1136,14 +1132,6 @@ int mca_btl_sm_ft_event(int state) {
else if(OPAL_CRS_RESTART == state ||
OPAL_CRS_RESTART_PRE == state) {
if( NULL != mca_btl_sm_component.sm_seg ) {
/* Add session directory */
opal_crs_base_cleanup_append(orte_process_info.job_session_dir, true);
tmp_dir = opal_dirname(orte_process_info.job_session_dir);
if( NULL != tmp_dir ) {
opal_crs_base_cleanup_append(tmp_dir, true);
free(tmp_dir);
tmp_dir = NULL;
}
/* Add shared memory file */
opal_crs_base_cleanup_append(mca_btl_sm_component.sm_seg->module_seg_path, false);
}

View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
# Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
# University Research and Technology
# Corporation. All rights reserved.
# Copyright (c) 2004-2005 The University of Tennessee and The University
@ -26,3 +26,4 @@ libmca_crcp_la_SOURCES += \
base/crcp_base_close.c \
base/crcp_base_select.c \
base/crcp_base_fns.c

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* Copyright (c) 2004-2005 The University of Tennessee and The University
@ -60,6 +60,12 @@ BEGIN_C_DECLS
*/
OMPI_DECLSPEC int ompi_crcp_base_close(void);
/**
* Quiesce Interface (For MPI Ext.)
*/
OMPI_DECLSPEC int ompi_crcp_base_quiesce_start(MPI_Info *info);
OMPI_DECLSPEC int ompi_crcp_base_quiesce_end(MPI_Info *info);
/**
* 'None' component functions
* These are to be used when no component is selected.
@ -72,6 +78,10 @@ BEGIN_C_DECLS
int ompi_crcp_base_module_init(void);
int ompi_crcp_base_module_finalize(void);
/* Quiesce Interface */
int ompi_crcp_base_none_quiesce_start(MPI_Info *info);
int ompi_crcp_base_none_quiesce_end(MPI_Info *info);
/* PML Interface */
ompi_crcp_base_pml_state_t* ompi_crcp_base_none_pml_enable( bool enable, ompi_crcp_base_pml_state_t* );

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2008 The Trustees of Indiana University.
* Copyright (c) 2004-2010 The Trustees of Indiana University.
* All rights reserved.
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
* All rights reserved.
@ -38,6 +38,7 @@
#include "ompi/mca/crcp/crcp.h"
#include "ompi/mca/crcp/base/base.h"
#include "ompi/mca/bml/base/base.h"
#include "ompi/info/info.h"
#include "ompi/mca/pml/pml.h"
#include "ompi/mca/pml/base/base.h"
#include "ompi/mca/pml/base/pml_base_request.h"
@ -92,6 +93,19 @@ int ompi_crcp_base_module_finalize(void)
return OMPI_SUCCESS;
}
/****************
* MPI Quiesce Interface
****************/
int ompi_crcp_base_none_quiesce_start(MPI_Info *info)
{
return OMPI_SUCCESS;
}
int ompi_crcp_base_none_quiesce_end(MPI_Info *info)
{
return OMPI_SUCCESS;
}
/****************
* PML Wrapper
****************/
@ -397,3 +411,24 @@ ompi_crcp_base_none_btl_ft_event(int state,
/********************
* Utility functions
********************/
/******************
* MPI Interface Functions
******************/
int ompi_crcp_base_quiesce_start(MPI_Info *info)
{
if( NULL != ompi_crcp.quiesce_start ) {
return ompi_crcp.quiesce_start(info);
} else {
return OMPI_SUCCESS;
}
}
int ompi_crcp_base_quiesce_end(MPI_Info *info)
{
if( NULL != ompi_crcp.quiesce_end ) {
return ompi_crcp.quiesce_end(info);
} else {
return OMPI_SUCCESS;
}
}

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2008 The Trustees of Indiana University.
* Copyright (c) 2004-2010 The Trustees of Indiana University.
* All rights reserved.
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
* All rights reserved.
@ -63,6 +63,10 @@ static ompi_crcp_base_module_t none_module = {
/** Finalization Function */
ompi_crcp_base_module_finalize,
/** Quiesce interface */
ompi_crcp_base_none_quiesce_start,
ompi_crcp_base_none_quiesce_end,
/** PML Wrapper */
ompi_crcp_base_none_pml_enable,

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2008 The Trustees of Indiana University.
* Copyright (c) 2004-2010 The Trustees of Indiana University.
* All rights reserved.
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
* All rights reserved.
@ -57,6 +57,12 @@ BEGIN_C_DECLS
int ompi_crcp_bkmrk_pml_init(void);
int ompi_crcp_bkmrk_pml_finalize(void);
/*
* Quiesce Interface
*/
int ompi_crcp_bkmrk_quiesce_start(MPI_Info *info);
int ompi_crcp_bkmrk_quiesce_end(MPI_Info *info);
END_C_DECLS
#endif /* MCA_CRCP_HOKE_EXPORT_H */

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2009 The Trustees of Indiana University.
* Copyright (c) 2004-2010 The Trustees of Indiana University.
* All rights reserved.
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
* All rights reserved.
@ -44,6 +44,10 @@ static ompi_crcp_base_module_t loc_module = {
/** Finalization Function */
ompi_crcp_bkmrk_module_finalize,
/** Quiesce interface */
ompi_crcp_bkmrk_quiesce_start,
ompi_crcp_bkmrk_quiesce_end,
/** PML Wrapper */
NULL, /* ompi_crcp_bkmrk_pml_enable, */
@ -131,6 +135,34 @@ int ompi_crcp_bkmrk_module_finalize(void)
return OMPI_SUCCESS;
}
int ompi_crcp_bkmrk_quiesce_start(MPI_Info *info)
{
OPAL_OUTPUT_VERBOSE((10, mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: quiesce_start(--)"));
#if 0
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_start(QUIESCE_TAG_CKPT)) ) {
;
}
return OMPI_SUCCESS;
#else
return OMPI_ERR_NOT_IMPLEMENTED;
#endif
}
int ompi_crcp_bkmrk_quiesce_end(MPI_Info *info)
{
OPAL_OUTPUT_VERBOSE((10, mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: quiesce_end(--)"));
#if 0
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_end(QUIESCE_TAG_CONTINUE) ) ) {
;
}
return OMPI_SUCCESS;
#else
return OMPI_ERR_NOT_IMPLEMENTED;
#endif
}
/******************
* Local functions
******************/

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2009 The Trustees of Indiana University.
* Copyright (c) 2004-2010 The Trustees of Indiana University.
* All rights reserved.
* Copyright (c) 2010 The University of Tennessee and The University
* of Tennessee Research Foundation. All rights
@ -2986,6 +2986,26 @@ int ompi_crcp_bkmrk_request_complete(struct ompi_request_t *request)
}
/**************** FT Event *****************/
int ompi_crcp_bkmrk_pml_quiesce_start(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag ) {
int ret, exit_status = OMPI_SUCCESS;
if( OMPI_SUCCESS != (ret = ft_event_coordinate_peers()) ) {
exit_status = ret;
}
return exit_status;
}
int ompi_crcp_bkmrk_pml_quiesce_end(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag ) {
int ret, exit_status = OMPI_SUCCESS;
if( OMPI_SUCCESS != (ret = ft_event_finalize_exchange() ) ) {
exit_status = ret;
}
return exit_status;
}
ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
int state,
ompi_crcp_base_pml_state_t* pml_state)
@ -3027,7 +3047,7 @@ ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
* When we return from this function we know that all of our
* channels have been flushed.
*/
if( OMPI_SUCCESS != (ret = ft_event_coordinate_peers()) ) {
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_start(QUIESCE_TAG_CKPT)) ) {
opal_output(mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: %s ft_event: Checkpoint Coordination Failed %d",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
@ -3060,7 +3080,7 @@ ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
first_continue_pass = !first_continue_pass;
/* Only finalize the Protocol after the PML has been rebuilt */
if( ompi_cr_continue_like_restart && first_continue_pass ) {
if( orte_cr_continue_like_restart && first_continue_pass ) {
goto DONE;
}
@ -3069,7 +3089,7 @@ ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
/*
* Finish the coord protocol
*/
if( OMPI_SUCCESS != (ret = ft_event_finalize_exchange() ) ) {
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_end(QUIESCE_TAG_CONTINUE) ) ) {
opal_output(mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: pml_ft_event: Checkpoint Finalization Failed %d",
ret);

View File

@ -1,5 +1,5 @@
/*
* Copyright (c) 2004-2007 The Trustees of Indiana University.
* Copyright (c) 2004-2010 The Trustees of Indiana University.
* All rights reserved.
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
* All rights reserved.
@ -116,6 +116,18 @@ BEGIN_C_DECLS
ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event
(int state, ompi_crcp_base_pml_state_t* pml_state);
enum ompi_crcp_bkmrk_pml_quiesce_tag_type_t {
QUIESCE_TAG_NONE = 0, /* 0 No tag specified */
QUIESCE_TAG_CKPT, /* 1 Prepare for checkpoint */
QUIESCE_TAG_CONTINUE, /* 2 Continue after a checkpoint */
QUIESCE_TAG_RESTART, /* 3 Restart from a checkpoint */
QUIESCE_TAG_UNKNOWN /* 4 Unknown */
};
typedef enum ompi_crcp_bkmrk_pml_quiesce_tag_type_t ompi_crcp_bkmrk_pml_quiesce_tag_type_t;
int ompi_crcp_bkmrk_pml_quiesce_start(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag );
int ompi_crcp_bkmrk_pml_quiesce_end(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag );
/*
* Request function
*/

View File

@ -61,6 +61,23 @@ typedef int (*ompi_crcp_base_module_init_fn_t)
typedef int (*ompi_crcp_base_module_finalize_fn_t)
(void);
/************************
* MPI Quiesce Interface
************************/
/**
* MPI_Quiesce_start component interface
*/
typedef int (*ompi_crcp_base_quiesce_start_fn_t)
(MPI_Info *info);
/**
* MPI_Quiesce_end component interface
*/
typedef int (*ompi_crcp_base_quiesce_end_fn_t)
(MPI_Info *info);
/************************
* PML Wrapper hooks
* PML Wrapper is the CRCPW PML component
@ -283,6 +300,10 @@ struct ompi_crcp_base_module_1_0_0_t {
/** Finalization Function */
ompi_crcp_base_module_finalize_fn_t crcp_finalize;
/**< MPI_Quiesce Interface Functions ******************/
ompi_crcp_base_quiesce_start_fn_t quiesce_start;
ompi_crcp_base_quiesce_end_fn_t quiesce_end;
/**< PML Wrapper Functions ****************************/
ompi_crcp_base_pml_enable_fn_t pml_enable;

View File

@ -32,6 +32,7 @@
#include "orte/util/proc_info.h"
#if OPAL_ENABLE_FT_CR == 1
#include "orte/mca/sstore/sstore.h"
#include "ompi/mca/mpool/base/base.h"
#include "ompi/runtime/ompi_cr.h"
#endif
@ -169,12 +170,12 @@ int mca_mpool_sm_ft_event(int state) {
asprintf( &file_name, "%s"OPAL_PATH_SEP"shared_mem_pool.%s",
orte_process_info.job_session_dir,
orte_process_info.nodename );
opal_crs_base_metadata_write_token(NULL, CRS_METADATA_TOUCH, file_name);
orte_sstore.set_attr(orte_sstore_handle_current, SSTORE_METADATA_LOCAL_TOUCH, file_name);
free(file_name);
file_name = NULL;
}
else if(OPAL_CRS_CONTINUE == state) {
if(ompi_cr_continue_like_restart) {
if(orte_cr_continue_like_restart) {
/* Find the sm module */
self_module = mca_mpool_base_module_lookup("sm");
self_sm_module = (mca_mpool_sm_module_t*) self_module;

View File

@ -691,7 +691,7 @@ int mca_pml_bfo_ft_event( int state )
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P2);
}
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
if( orte_cr_continue_like_restart && !first_continue_pass ) {
/*
* Get a list of processes
*/
@ -791,7 +791,7 @@ int mca_pml_bfo_ft_event( int state )
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P3);
}
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
if( orte_cr_continue_like_restart && !first_continue_pass ) {
/*
* Exchange the modex information once again.
* BTLs will have republished their modex information.

View File

@ -669,7 +669,7 @@ int mca_pml_csum_ft_event( int state )
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P2);
}
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
if( orte_cr_continue_like_restart && !first_continue_pass ) {
/*
* Get a list of processes
*/
@ -769,7 +769,7 @@ int mca_pml_csum_ft_event( int state )
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P3);
}
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
if( orte_cr_continue_like_restart && !first_continue_pass ) {
/*
* Exchange the modex information once again.
* BTLs will have republished their modex information.

View File

@ -638,7 +638,7 @@ int mca_pml_ob1_ft_event( int state )
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P2);
}
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
if( orte_cr_continue_like_restart && !first_continue_pass ) {
/*
* Get a list of processes
*/
@ -738,7 +738,7 @@ int mca_pml_ob1_ft_event( int state )
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P3);
}
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
if( orte_cr_continue_like_restart && !first_continue_pass ) {
/*
* Exchange the modex information once again.
* BTLs will have republished their modex information.

View File

@ -0,0 +1,38 @@
#
# Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
# University Research and Technology
# Corporation. All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
headers = \
mpiext_cr_c.h
sources = \
c/checkpoint.c \
c/restart.c \
c/migrate.c \
c/inc_register_callback.c \
c/quiesce_start.c \
c/quiesce_end.c \
c/quiesce_checkpoint.c \
c/self_register_checkpoint.c \
c/self_register_restart.c \
c/self_register_continue.c
lib = libext_mpiext_cr.la
lib_sources = $(sources)
extcomponentdir = $(pkglibdir)
noinst_LTLIBRARIES = $(lib)
libext_mpiext_cr_la_SOURCES = $(lib_sources)
libext_mpiext_cr_la_LDFLAGS = -module -avoid-version
ompidir = $(includedir)/openmpi/ompi/mpiext/cr
ompi_HEADERS = \
$(headers)

View File

@ -0,0 +1,88 @@
/*
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "ompi_config.h"
#include <stdio.h>
#include "ompi/mpi/c/bindings.h"
#include "ompi/info/info.h"
#include "ompi/runtime/params.h"
#include "ompi/communicator/communicator.h"
#include "orte/mca/snapc/snapc.h"
#include "ompi/mpiext/cr/mpiext_cr_c.h"
static const char FUNC_NAME[] = "OMPI_CR_Checkpoint";
#define HANDLE_SIZE_MAX 256
int OMPI_CR_Checkpoint(char **handle, int *seq, MPI_Info *info)
{
int ret = MPI_SUCCESS;
MPI_Comm comm = MPI_COMM_WORLD;
orte_snapc_base_request_op_t *datum = NULL;
int state = 0;
int my_rank;
/* argument checking */
if (MPI_PARAM_CHECK) {
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
}
/*
* Setup the data structure for the operation
*/
datum = OBJ_NEW(orte_snapc_base_request_op_t);
datum->event = ORTE_SNAPC_OP_CHECKPOINT;
datum->is_active = true;
MPI_Comm_rank(comm, &my_rank);
if( 0 == my_rank ) {
datum->leader = ORTE_PROC_MY_NAME->vpid;
} else {
datum->leader = -1; /* Unknown from non-root ranks */
}
/*
* All processes must make this call before it can start
*/
MPI_Barrier(comm);
/*
* Leader sends the request
*/
OPAL_CR_ENTER_LIBRARY();
ret = orte_snapc.request_op(datum);
if( OMPI_SUCCESS != ret ) {
OBJ_RELEASE(datum);
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
FUNC_NAME);
}
OPAL_CR_EXIT_LIBRARY();
/*
* Leader then sends out the commit message
*/
if( datum->leader == (int)ORTE_PROC_MY_NAME->vpid ) {
*handle = strdup(datum->global_handle);
*seq = datum->seq_num;
state = 0;
} else {
*handle = (char*)malloc(sizeof(char)*HANDLE_SIZE_MAX);
}
MPI_Bcast(&state, 1, MPI_INT, 0, comm);
MPI_Bcast(seq, 1, MPI_INT, 0, comm);
MPI_Bcast(*handle, HANDLE_SIZE_MAX, MPI_CHAR, 0, comm);
datum->is_active = false;
OBJ_RELEASE(datum);
return ret;
}

View File

@ -0,0 +1,39 @@
/*
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "ompi_config.h"
#include <stdio.h>
#include "ompi/mpi/c/bindings.h"
#include "opal/runtime/opal_cr.h"
#include "ompi/mpiext/cr/mpiext_cr_c.h"
#include "ompi/runtime/params.h"
#include "ompi/communicator/communicator.h"
#include "ompi/errhandler/errhandler.h"
static const char FUNC_NAME[] = "OMPI_CR_INC_register_callback";
int OMPI_CR_INC_register_callback(OMPI_CR_INC_callback_event_t event,
OMPI_CR_INC_callback_function function,
OMPI_CR_INC_callback_function *prev_function)
{
int rc;
if ( MPI_PARAM_CHECK ) {
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
}
OPAL_CR_ENTER_LIBRARY();
rc = opal_cr_user_inc_register_callback(event, function, prev_function);
OMPI_ERRHANDLER_RETURN(rc, MPI_COMM_WORLD, rc, FUNC_NAME);
}

120
ompi/mpiext/cr/c/migrate.c Normal file
View File

@ -0,0 +1,120 @@
/*
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "ompi_config.h"
#include <stdio.h>
#include "ompi/mpi/c/bindings.h"
#include "ompi/info/info.h"
#include "ompi/runtime/params.h"
#include "ompi/communicator/communicator.h"
#include "orte/mca/snapc/snapc.h"
#include "ompi/mpiext/cr/mpiext_cr_c.h"
static const char FUNC_NAME[] = "OMPI_CR_Migrate";
int OMPI_CR_Migrate(MPI_Comm comm, char *hostname, int rank, MPI_Info *info)
{
int ret = MPI_SUCCESS;
orte_snapc_base_request_op_t *datum = NULL;
int my_rank, my_size, i;
char loc_hostname[MPI_MAX_PROCESSOR_NAME];
int my_vpid;
int info_flag;
char info_value[6];
int my_off_node = (int)false;
/* argument checking */
if (MPI_PARAM_CHECK) {
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
}
/*
* Setup the data structure for the operation
*/
datum = OBJ_NEW(orte_snapc_base_request_op_t);
datum->event = ORTE_SNAPC_OP_MIGRATE;
datum->is_active = true;
MPI_Comm_rank(comm, &my_rank);
MPI_Comm_size(comm, &my_size);
if( 0 == my_rank ) {
datum->leader = ORTE_PROC_MY_NAME->vpid;
} else {
datum->leader = -1; /* Unknown from non-root ranks */
}
/*
* Gather all preferences to the root
*/
if( NULL == hostname ) {
loc_hostname[0] = '\0';
} else {
strncpy(loc_hostname, hostname, strlen(hostname));
loc_hostname[strlen(hostname)] = '\0';
}
my_vpid = (int) ORTE_PROC_MY_NAME->vpid;
if( 0 == my_rank ) {
datum->mig_num = my_size;
datum->mig_vpids = malloc(sizeof(int) * my_size);
datum->mig_host_pref = malloc(sizeof(char) * my_size * MPI_MAX_PROCESSOR_NAME);
datum->mig_vpid_pref = malloc(sizeof(int) * my_size);
datum->mig_off_node = malloc(sizeof(int) * my_size);
for( i = 0; i < my_size; ++i ) {
(datum->mig_vpids)[i] = 0;
(datum->mig_host_pref)[i][0] = '\0';
(datum->mig_vpid_pref)[i] = 0;
(datum->mig_off_node)[i] = (int)false;
}
}
my_off_node = (int)false;
if( NULL != info ) {
MPI_Info_get(*info, "CR_OFF_NODE", 5, info_value, &info_flag);
if( info_flag ) {
if( 0 == strncmp(info_value, "true", strlen("true")) ) {
my_off_node = (int)true;
}
}
}
MPI_Gather(&my_vpid, 1, MPI_INT,
(datum->mig_vpids), 1, MPI_INT, 0, comm);
MPI_Gather(loc_hostname, MPI_MAX_PROCESSOR_NAME, MPI_CHAR,
(datum->mig_host_pref), MPI_MAX_PROCESSOR_NAME, MPI_CHAR, 0, comm);
MPI_Gather(&my_vpid, 1, MPI_INT,
(datum->mig_vpid_pref), 1, MPI_INT, 0, comm);
MPI_Gather(&my_off_node, 1, MPI_INT,
(datum->mig_off_node), 1, MPI_INT, 0, comm);
/*
* Leader sends the request
*/
OPAL_CR_ENTER_LIBRARY();
ret = orte_snapc.request_op(datum);
if( OMPI_SUCCESS != ret ) {
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
FUNC_NAME);
}
OPAL_CR_EXIT_LIBRARY();
datum->is_active = false;
OBJ_RELEASE(datum);
/*
* All processes must sync before leaving
*/
MPI_Barrier(comm);
return ret;
}

View File

@ -0,0 +1,69 @@
/*
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "ompi_config.h"
#include <stdio.h>
#include "ompi/mpi/c/bindings.h"
#include "ompi/info/info.h"
#include "ompi/runtime/params.h"
#include "ompi/communicator/communicator.h"
#include "orte/mca/snapc/snapc.h"
#include "ompi/mpiext/cr/mpiext_cr_c.h"
static const char FUNC_NAME[] = "OMPI_CR_Quiesce_checkpoint";
int OMPI_CR_Quiesce_checkpoint(MPI_Comm commP, char **handle, int *seq, MPI_Info *info)
{
int ret = MPI_SUCCESS;
MPI_Comm comm = MPI_COMM_WORLD; /* Currently ignore provided comm */
orte_snapc_base_request_op_t *datum = NULL;
int my_rank;
/* argument checking */
if (MPI_PARAM_CHECK) {
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
}
/*
* Setup the data structure for the operation
*/
datum = OBJ_NEW(orte_snapc_base_request_op_t);
datum->event = ORTE_SNAPC_OP_QUIESCE_CHECKPOINT;
datum->is_active = true;
MPI_Comm_rank(comm, &my_rank);
if( 0 == my_rank ) {
datum->leader = ORTE_PROC_MY_NAME->vpid;
} else {
datum->leader = -1; /* Unknown from non-root ranks */
}
/*
* Since we are quiescent, then this is a local operation
*/
OPAL_CR_ENTER_LIBRARY();
ret = orte_snapc.request_op(datum);
/*ret = ompi_crcp_base_quiesce_start(info);*/
if( OMPI_SUCCESS != ret ) {
OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_OTHER,
FUNC_NAME);
}
OPAL_CR_EXIT_LIBRARY();
*handle = strdup(datum->global_handle);
*seq = datum->seq_num;
datum->is_active = false;
OBJ_RELEASE(datum);
return ret;
}

View File

@ -0,0 +1,74 @@
/*
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "ompi_config.h"
#include <stdio.h>
#include "ompi/mpi/c/bindings.h"
#include "ompi/info/info.h"
#include "ompi/runtime/params.h"
#include "ompi/communicator/communicator.h"
#include "orte/mca/snapc/snapc.h"
#include "ompi/mpiext/cr/mpiext_cr_c.h"
static const char FUNC_NAME[] = "OMPI_CR_Quiesce_end";
int OMPI_CR_Quiesce_end(MPI_Comm commP, MPI_Info *info)
{
int ret = MPI_SUCCESS;
MPI_Comm comm = MPI_COMM_WORLD; /* Currently ignore provided comm */
orte_snapc_base_request_op_t *datum = NULL;
int my_rank;
/* argument checking */
if (MPI_PARAM_CHECK) {
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
}
/*
* Setup the data structure for the operation
*/
datum = OBJ_NEW(orte_snapc_base_request_op_t);
datum->event = ORTE_SNAPC_OP_QUIESCE_END;
datum->is_active = true;
MPI_Comm_rank(comm, &my_rank);
if( 0 == my_rank ) {
datum->leader = ORTE_PROC_MY_NAME->vpid;
} else {
datum->leader = -1; /* Unknown from non-root ranks */
}
/*
* Leader sends the request
*/
OPAL_CR_ENTER_LIBRARY();
ret = orte_snapc.request_op(datum);
/*ret = ompi_crcp_base_quiesce_end(info);*/
if( OMPI_SUCCESS != ret ) {
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
FUNC_NAME);
}
OPAL_CR_EXIT_LIBRARY();
/*
* All processes must make this call before it can complete
*/
MPI_Barrier(comm);
/*
* (Old) info logic
*/
/*cur_datum.epoch = -1;*/
return ret;
}

View File

@ -0,0 +1,210 @@
/*
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "ompi_config.h"
#include <stdio.h>
#include "ompi/mpi/c/bindings.h"
#include "ompi/info/info.h"
#include "ompi/runtime/params.h"
#include "ompi/communicator/communicator.h"
#include "orte/mca/snapc/snapc.h"
#include "ompi/mpiext/cr/mpiext_cr_c.h"
static const char FUNC_NAME[] = "OMPI_CR_Quiesce_start";
int OMPI_CR_Quiesce_start(MPI_Comm commP, MPI_Info *info)
{
int ret = MPI_SUCCESS;
MPI_Comm comm = MPI_COMM_WORLD; /* Currently ignore provided comm */
orte_snapc_base_request_op_t *datum = NULL;
int my_rank;
/* argument checking */
if (MPI_PARAM_CHECK) {
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
}
/*
* Setup the data structure for the operation
*/
datum = OBJ_NEW(orte_snapc_base_request_op_t);
datum->event = ORTE_SNAPC_OP_QUIESCE_START;
datum->is_active = true;
MPI_Comm_rank(comm, &my_rank);
if( 0 == my_rank ) {
datum->leader = ORTE_PROC_MY_NAME->vpid;
} else {
datum->leader = -1; /* Unknown from non-root ranks */
}
/*
* All processes must make this call before it can start
*/
MPI_Barrier(comm);
/*
* Leader sends the request
*/
OPAL_CR_ENTER_LIBRARY();
ret = orte_snapc.request_op(datum);
/*ret = ompi_crcp_base_quiesce_start(info);*/
if( OMPI_SUCCESS != ret ) {
OBJ_RELEASE(datum);
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
FUNC_NAME);
}
OPAL_CR_EXIT_LIBRARY();
datum->is_active = false;
OBJ_RELEASE(datum);
/*
* (Old) info logic
*/
/*ompi_info_set((ompi_info_t*)*info, "target", cur_datum.target_dir);*/
return ret;
}
/*****************
* Local Functions
******************/
#if 0
/* Info keys:
*
* - crs:
* none = (Default) No CRS Service
* default = Whatever CRS service MPI chooses
* blcr = BLCR
* self = app level callbacks
*
* - cmdline:
* Command line to restart the process with.
* If empty, the user must manually enter it
*
* - target:
* Absolute path to the target directory.
*
* - handle:
* first = Earliest checkpoint directory available
* last = Most recent checkpoint directory available
* [global:local] = handle provided by the MPI library
*
* - restarting:
* 0 = not restarting
* 1 = restarting
*
* - checkpointing:
* 0 = No need to prepare for checkpointing
* 1 = MPI should prepare for checkpointing
*
* - inflight:
* default = message
* message = Drain inflight messages at the message level
* network = Drain inflight messages at the network level (if possible)
*
* - user_space_mem:
* 0 = Memory does not need to be managed
* 1 = Memory must be in user space (i.e., not on network card
*
*/
static int extract_info_into_datum(ompi_info_t *info, orte_snapc_base_quiesce_t *datum)
{
int info_flag = false;
int max_crs_len = 32;
bool info_bool = false;
char *info_char = NULL;
info_char = (char *) malloc(sizeof(char) * (OPAL_PATH_MAX+1));
/*
* Key: crs
*/
ompi_info_get(info, "crs", max_crs_len, info_char, &info_flag);
if( info_flag) {
datum->crs_name = strdup(info_char);
}
/*
* Key: cmdline
*/
ompi_info_get(info, "cmdline", OPAL_PATH_MAX, info_char, &info_flag);
if( info_flag) {
datum->cmdline = strdup(info_char);
}
/*
* Key: handle
*/
ompi_info_get(info, "handle", OPAL_PATH_MAX, info_char, &info_flag);
if( info_flag) {
datum->handle = strdup(info_char);
}
/*
* Key: target
*/
ompi_info_get(info, "target", OPAL_PATH_MAX, info_char, &info_flag);
if( info_flag) {
datum->target_dir = strdup(info_char);
}
/*
* Key: restarting
*/
ompi_info_get_bool(info, "restarting", &info_bool, &info_flag);
if( info_flag ) {
datum->restarting = info_bool;
} else {
datum->restarting = false;
}
/*
* Key: checkpointing
*/
ompi_info_get_bool(info, "checkpointing", &info_bool, &info_flag);
if( info_flag ) {
datum->checkpointing = info_bool;
} else {
datum->checkpointing = false;
}
/*
* Display all values
*/
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: %s extract_info: Info('crs' = '%s')",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
(NULL == datum->crs_name ? "Default (none)" : datum->crs_name)));
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: %s extract_info: Info('cmdline' = '%s')",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
(NULL == datum->cmdline ? "Default ()" : datum->cmdline)));
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: %s extract_info: Info('checkpointing' = '%c')",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
(datum->checkpointing ? 'T' : 'F')));
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
"crcp:bkmrk: %s extract_info: Info('restarting' = '%c')",
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),