A number of C/R enhancements per RFC below:
http://www.open-mpi.org/community/lists/devel/2010/07/8240.php Documentation: http://osl.iu.edu/research/ft/ Major Changes: -------------- * Added C/R-enabled Debugging support. Enabled with the --enable-crdebug flag. See the following website for more information: http://osl.iu.edu/research/ft/crdebug/ * Added Stable Storage (SStore) framework for checkpoint storage * 'central' component does a direct to central storage save * 'stage' component stages checkpoints to central storage while the application continues execution. * 'stage' supports offline compression of checkpoints before moving (sstore_stage_compress) * 'stage' supports local caching of checkpoints to improve automatic recovery (sstore_stage_caching) * Added Compression (compress) framework to support * Add two new ErrMgr recovery policies * {{{crmig}}} C/R Process Migration * {{{autor}}} C/R Automatic Recovery * Added the {{{ompi-migrate}}} command line tool to support the {{{crmig}}} ErrMgr component * Added CR MPI Ext functions (enable them with {{{--enable-mpi-ext=cr}}} configure option) * {{{OMPI_CR_Checkpoint}}} (Fixes trac:2342) * {{{OMPI_CR_Restart}}} * {{{OMPI_CR_Migrate}}} (may need some more work for mapping rules) * {{{OMPI_CR_INC_register_callback}}} (Fixes trac:2192) * {{{OMPI_CR_Quiesce_start}}} * {{{OMPI_CR_Quiesce_checkpoint}}} * {{{OMPI_CR_Quiesce_end}}} * {{{OMPI_CR_self_register_checkpoint_callback}}} * {{{OMPI_CR_self_register_restart_callback}}} * {{{OMPI_CR_self_register_continue_callback}}} * The ErrMgr predicted_fault() interface has been changed to take an opal_list_t of ErrMgr defined types. This will allow us to better support a wider range of fault prediction services in the future. * Add a progress meter to: * FileM rsh (filem_rsh_process_meter) * SnapC full (snapc_full_progress_meter) * SStore stage (sstore_stage_progress_meter) * Added 2 new command line options to ompi-restart * --showme : Display the full command line that would have been exec'ed. * --mpirun_opts : Command line options to pass directly to mpirun. (Fixes trac:2413) * Deprecated some MCA params: * crs_base_snapshot_dir deprecated, use sstore_stage_local_snapshot_dir * snapc_base_global_snapshot_dir deprecated, use sstore_base_global_snapshot_dir * snapc_base_global_shared deprecated, use sstore_stage_global_is_shared * snapc_base_store_in_place deprecated, replaced with different components of SStore * snapc_base_global_snapshot_ref deprecated, use sstore_base_global_snapshot_ref * snapc_base_establish_global_snapshot_dir deprecated, never well supported * snapc_full_skip_filem deprecated, use sstore_stage_skip_filem Minor Changes: -------------- * Fixes trac:1924 : {{{ompi-restart}}} now recognizes path prefixed checkpoint handles and does the right thing. * Fixes trac:2097 : {{{ompi-info}}} should now report all available CRS components * Fixes trac:2161 : Manual checkpoint movement. A user can 'mv' a checkpoint directory from the original location to another and still restart from it. * Fixes trac:2208 : Honor various TMPDIR varaibles instead of forcing {{{/tmp}}} * Move {{{ompi_cr_continue_like_restart}}} to {{{orte_cr_continue_like_restart}}} to be more flexible in where this should be set. * opal_crs_base_metadata_write* functions have been moved to SStore to support a wider range of metadata handling functionality. * Cleanup the CRS framework and components to work with the SStore framework. * Cleanup the SnapC framework and components to work with the SStore framework (cleans up these code paths considerably). * Add 'quiesce' hook to CRCP for a future enhancement. * We now require a BLCR version that supports {{{cr_request_file()}}} or {{{cr_request_checkpoint()}}} in order to make the code more maintainable. Note that {{{cr_request_file}}} has been deprecated since 0.7.0, so we prefer to use {{{cr_request_checkpoint()}}}. * Add optional application level INC callbacks (registered through the CR MPI Ext interface). * Increase the {{{opal_cr_thread_sleep_wait}}} parameter to 1000 microseconds to make the C/R thread less aggressive. * {{{opal-restart}}} now looks for cache directories before falling back on stable storage when asked. * {{{opal-restart}}} also support local decompression before restarting * {{{orte-checkpoint}}} now uses the SStore framework to work with the metadata * {{{orte-restart}}} now uses the SStore framework to work with the metadata * Remove the {{{orte-restart}}} preload option. This was removed since the user only needs to select the 'stage' component in order to support this functionality. * Since the '-am' parameter is saved in the metadata, {{{ompi-restart}}} no longer hard codes {{{-am ft-enable-cr}}}. * Fix {{{hnp}}} ErrMgr so that if a previous component in the stack has 'fixed' the problem, then it should be skipped. * Make sure to decrement the number of 'num_local_procs' in the orted when one goes away. * odls now checks the SStore framework to see if it needs to load any checkpoint files before launching (to support 'stage'). This separates the SStore logic from the --preload-[binary|files] options. * Add unique IDs to the named pipes established between the orted and the app in SnapC. This is to better support migration and automatic recovery activities. * Improve the checks for 'already checkpointing' error path. * A a recovery output timer, to show how long it takes to restart a job * Do a better job of cleaning up the old session directory on restart. * Add a local module to the autor and crmig ErrMgr components. These small modules prevent the 'orted' component from attempting a local recovery (Which does not work for MPI apps at the moment) * Add a fix for bounding the checkpointable region between MPI_Init and MPI_Finalize. This commit was SVN r23587. The following Trac tickets were found above: Ticket 1924 --> https://svn.open-mpi.org/trac/ompi/ticket/1924 Ticket 2097 --> https://svn.open-mpi.org/trac/ompi/ticket/2097 Ticket 2161 --> https://svn.open-mpi.org/trac/ompi/ticket/2161 Ticket 2192 --> https://svn.open-mpi.org/trac/ompi/ticket/2192 Ticket 2208 --> https://svn.open-mpi.org/trac/ompi/ticket/2208 Ticket 2342 --> https://svn.open-mpi.org/trac/ompi/ticket/2342 Ticket 2413 --> https://svn.open-mpi.org/trac/ompi/ticket/2413
This commit is contained in:
parent
9fff01704f
commit
e12ca48cd9
@ -1,5 +1,5 @@
|
||||
#
|
||||
# Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
||||
# Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
# University Research and Technology
|
||||
# Corporation. All rights reserved.
|
||||
# Copyright (c) 2004-2005 The University of Tennessee and The University
|
||||
@ -22,7 +22,9 @@ amca_paramdir = $(AMCA_PARAM_SETS_DIR)
|
||||
dist_amca_param_DATA = amca-param-sets/example.conf
|
||||
|
||||
if WANT_FT
|
||||
dist_amca_param_DATA += amca-param-sets/ft-enable-cr
|
||||
dist_amca_param_DATA += \
|
||||
amca-param-sets/ft-enable-cr \
|
||||
amca-param-sets/ft-enable-cr-recovery
|
||||
endif
|
||||
|
||||
EXTRA_DIST = \
|
||||
|
@ -1,5 +1,5 @@
|
||||
#
|
||||
# Copyright (c) 2008-2009 The Trustees of Indiana University and Indiana
|
||||
# Copyright (c) 2008-2010 The Trustees of Indiana University and Indiana
|
||||
# University Research and Technology
|
||||
# Corporation. All rights reserved.
|
||||
#
|
||||
@ -37,7 +37,6 @@ opal_cr_use_thread=1
|
||||
#
|
||||
rml_wrapper=ftrm
|
||||
snapc=full
|
||||
#filem=rsh
|
||||
|
||||
#
|
||||
# OMPI Parameters
|
||||
|
82
contrib/amca-param-sets/ft-enable-cr-recovery
Normal file
82
contrib/amca-param-sets/ft-enable-cr-recovery
Normal file
@ -0,0 +1,82 @@
|
||||
#
|
||||
# Copyright (c) 2009-2010 The Trustees of Indiana University and Indiana
|
||||
# University Research and Technology
|
||||
# Corporation. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
#
|
||||
# $HEADER$
|
||||
#
|
||||
# An Aggregate MCA Parameter Set to enable checkpoint/restart capabilities
|
||||
# for a job.
|
||||
#
|
||||
# Usage:
|
||||
# shell$ mpirun -am ft-enable-cr ./app
|
||||
#
|
||||
|
||||
#
|
||||
# OPAL Parameters
|
||||
# - Turn off OPAL only checkpointing
|
||||
# - Select only checkpoint ready components
|
||||
# - Enable Additional FT infrastructure
|
||||
# - Auto-select OPAL CRS component
|
||||
# - If available, use the FT Thread (Default)
|
||||
#
|
||||
opal_cr_allow_opal_only=0
|
||||
mca_base_component_distill_checkpoint_ready=1
|
||||
ft_cr_enabled=1
|
||||
crs=
|
||||
opal_cr_use_thread=1
|
||||
|
||||
#
|
||||
# ORTE Parameters
|
||||
# - Wrap the RML
|
||||
# - Use the 'full' Snapshot Coordinator
|
||||
# - Use the 'cm' routed component. It is the only one that is currently able to
|
||||
# handle process and daemon loss.
|
||||
#
|
||||
rml_wrapper=ftrm
|
||||
snapc=full
|
||||
routed=cm
|
||||
|
||||
#
|
||||
# OMPI Parameters
|
||||
# - Wrap the PML
|
||||
# - Use a Bookmark Exchange Fully Coordinated Checkpoint/Restart Coordination Protocol
|
||||
#
|
||||
pml_wrapper=crcpw
|
||||
crcp=bkmrk
|
||||
|
||||
#
|
||||
# Temporary fix to force the event engine to use poll to behave well with BLCR
|
||||
#
|
||||
opal_event_include=poll
|
||||
|
||||
#
|
||||
# We currently only support the following options to the OpenIB BTL
|
||||
# Future development will attempt to eliminate many of these restrictions
|
||||
#
|
||||
btl_openib_want_fork_support=1
|
||||
btl_openib_use_async_event_thread=0
|
||||
btl_openib_use_eager_rdma=0
|
||||
btl_openib_cpc_include=oob
|
||||
|
||||
# Enable SIGTSTP/SIGCONT capability
|
||||
# killall -TSTP mpirun
|
||||
# killall -CONT mpirun
|
||||
orte_forward_job_control=1
|
||||
|
||||
#
|
||||
# Use the C/R Error Management and Recovery Service
|
||||
#
|
||||
orte_enable_recovery=1
|
||||
orte_max_global_restarts=10
|
||||
errmgr_crmig_enable=1
|
||||
errmgr_autor_enable=1
|
||||
|
||||
#
|
||||
# Additional constraints to be lifted in the future
|
||||
#
|
||||
plm=rsh
|
||||
rmaps=resilient
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* Copyright (c) 2004-2007 The University of Tennessee and The University
|
||||
@ -54,7 +54,7 @@ int mca_bml_r2_ft_event(int state)
|
||||
first_continue_pass = !first_continue_pass;
|
||||
|
||||
/* Since nothing in Checkpoint, we are fine here (unless required by BTL) */
|
||||
if( ompi_cr_continue_like_restart && !first_continue_pass) {
|
||||
if( orte_cr_continue_like_restart && !first_continue_pass) {
|
||||
procs = ompi_proc_all(&num_procs);
|
||||
if(NULL == procs) {
|
||||
return OMPI_ERR_OUT_OF_RESOURCE;
|
||||
@ -136,7 +136,7 @@ int mca_bml_r2_ft_event(int state)
|
||||
}
|
||||
else if(OPAL_CRS_CONTINUE == state) {
|
||||
/* Matches OPAL_CRS_RESTART_PRE */
|
||||
if( ompi_cr_continue_like_restart && first_continue_pass) {
|
||||
if( orte_cr_continue_like_restart && first_continue_pass) {
|
||||
if( OMPI_SUCCESS != (ret = mca_bml_r2_finalize()) ) {
|
||||
opal_output(0, "bml:r2: ft_event(Restart): Failed to finalize BML framework\n");
|
||||
return ret;
|
||||
@ -147,7 +147,7 @@ int mca_bml_r2_ft_event(int state)
|
||||
}
|
||||
}
|
||||
/* Matches OPAL_CRS_RESTART */
|
||||
else if( ompi_cr_continue_like_restart && !first_continue_pass ) {
|
||||
else if( orte_cr_continue_like_restart && !first_continue_pass ) {
|
||||
/*
|
||||
* Barrier to make all processes have been successfully restarted before
|
||||
* we try to remove some restart only files.
|
||||
@ -157,10 +157,6 @@ int mca_bml_r2_ft_event(int state)
|
||||
return ret;
|
||||
}
|
||||
|
||||
opal_output_verbose(10, ompi_cr_output,
|
||||
"bml:r2: ft_event(Restart): Cleanup restart files\n");
|
||||
opal_crs_base_cleanup_flush();
|
||||
|
||||
/*
|
||||
* Re-open the BTL framework to get the full list of components.
|
||||
*/
|
||||
@ -234,10 +230,6 @@ int mca_bml_r2_ft_event(int state)
|
||||
return ret;
|
||||
}
|
||||
|
||||
opal_output_verbose(10, ompi_cr_output,
|
||||
"bml:r2: ft_event(Restart): Cleanup restart files\n");
|
||||
opal_crs_base_cleanup_flush();
|
||||
|
||||
/*
|
||||
* Re-open the BTL framework to get the full list of components.
|
||||
* - but first clear the MCA value that was there
|
||||
|
@ -641,7 +641,7 @@ int mca_btl_mx_ft_event(int state) {
|
||||
* kernel: blcr: thaw_threads returned error, aborting. -1
|
||||
* JJH: It may be possible to, instead of restarting the entire driver, just reconnect endpoints
|
||||
*/
|
||||
ompi_cr_continue_like_restart = true;
|
||||
orte_cr_continue_like_restart = true;
|
||||
|
||||
for( i = 0; i < mca_btl_mx_component.mx_num_btls; i++ ) {
|
||||
mx_btl = mca_btl_mx_component.mx_btls[i];
|
||||
|
@ -1735,7 +1735,7 @@ int mca_btl_openib_ft_event(int state) {
|
||||
if(OPAL_CRS_CHECKPOINT == state) {
|
||||
/* Continue must reconstruct the routes (including modex), since we
|
||||
* have to tear down the devices completely. */
|
||||
ompi_cr_continue_like_restart = true;
|
||||
orte_cr_continue_like_restart = true;
|
||||
|
||||
/*
|
||||
* To keep the node from crashing we need to call ibv_close_device
|
||||
|
@ -52,6 +52,7 @@
|
||||
#if OPAL_ENABLE_FT_CR == 1
|
||||
#include "opal/mca/crs/base/base.h"
|
||||
#include "opal/util/basename.h"
|
||||
#include "orte/mca/sstore/sstore.h"
|
||||
#include "ompi/runtime/ompi_cr.h"
|
||||
#endif
|
||||
|
||||
@ -1099,8 +1100,6 @@ int mca_btl_sm_ft_event(int state) {
|
||||
}
|
||||
#else
|
||||
int mca_btl_sm_ft_event(int state) {
|
||||
char * tmp_dir = NULL;
|
||||
|
||||
/* Notify mpool */
|
||||
if( NULL != mca_btl_sm_component.sm_mpool &&
|
||||
NULL != mca_btl_sm_component.sm_mpool->mpool_ft_event) {
|
||||
@ -1114,17 +1113,14 @@ int mca_btl_sm_ft_event(int state) {
|
||||
* for these old file handles. The restart procedure will make sure
|
||||
* these files get cleaned up appropriately.
|
||||
*/
|
||||
opal_crs_base_metadata_write_token(NULL, CRS_METADATA_TOUCH, mca_btl_sm_component.sm_seg->module_seg_path);
|
||||
|
||||
/* Record the job session directory */
|
||||
opal_crs_base_metadata_write_token(NULL, CRS_METADATA_MKDIR, orte_process_info.job_session_dir);
|
||||
orte_sstore.set_attr(orte_sstore_handle_current,
|
||||
SSTORE_METADATA_LOCAL_TOUCH,
|
||||
mca_btl_sm_component.sm_seg->module_seg_path);
|
||||
}
|
||||
}
|
||||
else if(OPAL_CRS_CONTINUE == state) {
|
||||
if( ompi_cr_continue_like_restart ) {
|
||||
if( orte_cr_continue_like_restart ) {
|
||||
if( NULL != mca_btl_sm_component.sm_seg ) {
|
||||
/* Do not Add session directory on continue */
|
||||
|
||||
/* Add shared memory file */
|
||||
opal_crs_base_cleanup_append(mca_btl_sm_component.sm_seg->module_seg_path, false);
|
||||
}
|
||||
@ -1136,14 +1132,6 @@ int mca_btl_sm_ft_event(int state) {
|
||||
else if(OPAL_CRS_RESTART == state ||
|
||||
OPAL_CRS_RESTART_PRE == state) {
|
||||
if( NULL != mca_btl_sm_component.sm_seg ) {
|
||||
/* Add session directory */
|
||||
opal_crs_base_cleanup_append(orte_process_info.job_session_dir, true);
|
||||
tmp_dir = opal_dirname(orte_process_info.job_session_dir);
|
||||
if( NULL != tmp_dir ) {
|
||||
opal_crs_base_cleanup_append(tmp_dir, true);
|
||||
free(tmp_dir);
|
||||
tmp_dir = NULL;
|
||||
}
|
||||
/* Add shared memory file */
|
||||
opal_crs_base_cleanup_append(mca_btl_sm_component.sm_seg->module_seg_path, false);
|
||||
}
|
||||
|
@ -1,5 +1,5 @@
|
||||
#
|
||||
# Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
||||
# Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
# University Research and Technology
|
||||
# Corporation. All rights reserved.
|
||||
# Copyright (c) 2004-2005 The University of Tennessee and The University
|
||||
@ -26,3 +26,4 @@ libmca_crcp_la_SOURCES += \
|
||||
base/crcp_base_close.c \
|
||||
base/crcp_base_select.c \
|
||||
base/crcp_base_fns.c
|
||||
|
||||
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The University of Tennessee and The University
|
||||
@ -60,6 +60,12 @@ BEGIN_C_DECLS
|
||||
*/
|
||||
OMPI_DECLSPEC int ompi_crcp_base_close(void);
|
||||
|
||||
/**
|
||||
* Quiesce Interface (For MPI Ext.)
|
||||
*/
|
||||
OMPI_DECLSPEC int ompi_crcp_base_quiesce_start(MPI_Info *info);
|
||||
OMPI_DECLSPEC int ompi_crcp_base_quiesce_end(MPI_Info *info);
|
||||
|
||||
/**
|
||||
* 'None' component functions
|
||||
* These are to be used when no component is selected.
|
||||
@ -72,6 +78,10 @@ BEGIN_C_DECLS
|
||||
int ompi_crcp_base_module_init(void);
|
||||
int ompi_crcp_base_module_finalize(void);
|
||||
|
||||
/* Quiesce Interface */
|
||||
int ompi_crcp_base_none_quiesce_start(MPI_Info *info);
|
||||
int ompi_crcp_base_none_quiesce_end(MPI_Info *info);
|
||||
|
||||
/* PML Interface */
|
||||
ompi_crcp_base_pml_state_t* ompi_crcp_base_none_pml_enable( bool enable, ompi_crcp_base_pml_state_t* );
|
||||
|
||||
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2008 The Trustees of Indiana University.
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
|
||||
* All rights reserved.
|
||||
@ -38,6 +38,7 @@
|
||||
#include "ompi/mca/crcp/crcp.h"
|
||||
#include "ompi/mca/crcp/base/base.h"
|
||||
#include "ompi/mca/bml/base/base.h"
|
||||
#include "ompi/info/info.h"
|
||||
#include "ompi/mca/pml/pml.h"
|
||||
#include "ompi/mca/pml/base/base.h"
|
||||
#include "ompi/mca/pml/base/pml_base_request.h"
|
||||
@ -92,6 +93,19 @@ int ompi_crcp_base_module_finalize(void)
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
/****************
|
||||
* MPI Quiesce Interface
|
||||
****************/
|
||||
int ompi_crcp_base_none_quiesce_start(MPI_Info *info)
|
||||
{
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
int ompi_crcp_base_none_quiesce_end(MPI_Info *info)
|
||||
{
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
/****************
|
||||
* PML Wrapper
|
||||
****************/
|
||||
@ -397,3 +411,24 @@ ompi_crcp_base_none_btl_ft_event(int state,
|
||||
/********************
|
||||
* Utility functions
|
||||
********************/
|
||||
|
||||
/******************
|
||||
* MPI Interface Functions
|
||||
******************/
|
||||
int ompi_crcp_base_quiesce_start(MPI_Info *info)
|
||||
{
|
||||
if( NULL != ompi_crcp.quiesce_start ) {
|
||||
return ompi_crcp.quiesce_start(info);
|
||||
} else {
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
}
|
||||
|
||||
int ompi_crcp_base_quiesce_end(MPI_Info *info)
|
||||
{
|
||||
if( NULL != ompi_crcp.quiesce_end ) {
|
||||
return ompi_crcp.quiesce_end(info);
|
||||
} else {
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
}
|
||||
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2008 The Trustees of Indiana University.
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
|
||||
* All rights reserved.
|
||||
@ -63,6 +63,10 @@ static ompi_crcp_base_module_t none_module = {
|
||||
/** Finalization Function */
|
||||
ompi_crcp_base_module_finalize,
|
||||
|
||||
/** Quiesce interface */
|
||||
ompi_crcp_base_none_quiesce_start,
|
||||
ompi_crcp_base_none_quiesce_end,
|
||||
|
||||
/** PML Wrapper */
|
||||
ompi_crcp_base_none_pml_enable,
|
||||
|
||||
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2008 The Trustees of Indiana University.
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
|
||||
* All rights reserved.
|
||||
@ -57,6 +57,12 @@ BEGIN_C_DECLS
|
||||
int ompi_crcp_bkmrk_pml_init(void);
|
||||
int ompi_crcp_bkmrk_pml_finalize(void);
|
||||
|
||||
/*
|
||||
* Quiesce Interface
|
||||
*/
|
||||
int ompi_crcp_bkmrk_quiesce_start(MPI_Info *info);
|
||||
int ompi_crcp_bkmrk_quiesce_end(MPI_Info *info);
|
||||
|
||||
END_C_DECLS
|
||||
|
||||
#endif /* MCA_CRCP_HOKE_EXPORT_H */
|
||||
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2009 The Trustees of Indiana University.
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
|
||||
* All rights reserved.
|
||||
@ -44,6 +44,10 @@ static ompi_crcp_base_module_t loc_module = {
|
||||
/** Finalization Function */
|
||||
ompi_crcp_bkmrk_module_finalize,
|
||||
|
||||
/** Quiesce interface */
|
||||
ompi_crcp_bkmrk_quiesce_start,
|
||||
ompi_crcp_bkmrk_quiesce_end,
|
||||
|
||||
/** PML Wrapper */
|
||||
NULL, /* ompi_crcp_bkmrk_pml_enable, */
|
||||
|
||||
@ -131,6 +135,34 @@ int ompi_crcp_bkmrk_module_finalize(void)
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
int ompi_crcp_bkmrk_quiesce_start(MPI_Info *info)
|
||||
{
|
||||
OPAL_OUTPUT_VERBOSE((10, mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: quiesce_start(--)"));
|
||||
#if 0
|
||||
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_start(QUIESCE_TAG_CKPT)) ) {
|
||||
;
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
#else
|
||||
return OMPI_ERR_NOT_IMPLEMENTED;
|
||||
#endif
|
||||
}
|
||||
|
||||
int ompi_crcp_bkmrk_quiesce_end(MPI_Info *info)
|
||||
{
|
||||
OPAL_OUTPUT_VERBOSE((10, mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: quiesce_end(--)"));
|
||||
#if 0
|
||||
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_end(QUIESCE_TAG_CONTINUE) ) ) {
|
||||
;
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
#else
|
||||
return OMPI_ERR_NOT_IMPLEMENTED;
|
||||
#endif
|
||||
}
|
||||
|
||||
/******************
|
||||
* Local functions
|
||||
******************/
|
||||
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2009 The Trustees of Indiana University.
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2010 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
@ -2986,6 +2986,26 @@ int ompi_crcp_bkmrk_request_complete(struct ompi_request_t *request)
|
||||
}
|
||||
|
||||
/**************** FT Event *****************/
|
||||
int ompi_crcp_bkmrk_pml_quiesce_start(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag ) {
|
||||
int ret, exit_status = OMPI_SUCCESS;
|
||||
|
||||
if( OMPI_SUCCESS != (ret = ft_event_coordinate_peers()) ) {
|
||||
exit_status = ret;
|
||||
}
|
||||
|
||||
return exit_status;
|
||||
}
|
||||
|
||||
int ompi_crcp_bkmrk_pml_quiesce_end(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag ) {
|
||||
int ret, exit_status = OMPI_SUCCESS;
|
||||
|
||||
if( OMPI_SUCCESS != (ret = ft_event_finalize_exchange() ) ) {
|
||||
exit_status = ret;
|
||||
}
|
||||
|
||||
return exit_status;
|
||||
}
|
||||
|
||||
ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
|
||||
int state,
|
||||
ompi_crcp_base_pml_state_t* pml_state)
|
||||
@ -3027,7 +3047,7 @@ ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
|
||||
* When we return from this function we know that all of our
|
||||
* channels have been flushed.
|
||||
*/
|
||||
if( OMPI_SUCCESS != (ret = ft_event_coordinate_peers()) ) {
|
||||
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_start(QUIESCE_TAG_CKPT)) ) {
|
||||
opal_output(mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: %s ft_event: Checkpoint Coordination Failed %d",
|
||||
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
||||
@ -3060,7 +3080,7 @@ ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
|
||||
first_continue_pass = !first_continue_pass;
|
||||
|
||||
/* Only finalize the Protocol after the PML has been rebuilt */
|
||||
if( ompi_cr_continue_like_restart && first_continue_pass ) {
|
||||
if( orte_cr_continue_like_restart && first_continue_pass ) {
|
||||
goto DONE;
|
||||
}
|
||||
|
||||
@ -3069,7 +3089,7 @@ ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event(
|
||||
/*
|
||||
* Finish the coord protocol
|
||||
*/
|
||||
if( OMPI_SUCCESS != (ret = ft_event_finalize_exchange() ) ) {
|
||||
if( OMPI_SUCCESS != (ret = ompi_crcp_bkmrk_pml_quiesce_end(QUIESCE_TAG_CONTINUE) ) ) {
|
||||
opal_output(mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: pml_ft_event: Checkpoint Finalization Failed %d",
|
||||
ret);
|
||||
|
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2007 The Trustees of Indiana University.
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
|
||||
* All rights reserved.
|
||||
@ -116,6 +116,18 @@ BEGIN_C_DECLS
|
||||
ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_ft_event
|
||||
(int state, ompi_crcp_base_pml_state_t* pml_state);
|
||||
|
||||
enum ompi_crcp_bkmrk_pml_quiesce_tag_type_t {
|
||||
QUIESCE_TAG_NONE = 0, /* 0 No tag specified */
|
||||
QUIESCE_TAG_CKPT, /* 1 Prepare for checkpoint */
|
||||
QUIESCE_TAG_CONTINUE, /* 2 Continue after a checkpoint */
|
||||
QUIESCE_TAG_RESTART, /* 3 Restart from a checkpoint */
|
||||
QUIESCE_TAG_UNKNOWN /* 4 Unknown */
|
||||
};
|
||||
typedef enum ompi_crcp_bkmrk_pml_quiesce_tag_type_t ompi_crcp_bkmrk_pml_quiesce_tag_type_t;
|
||||
|
||||
int ompi_crcp_bkmrk_pml_quiesce_start(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag );
|
||||
int ompi_crcp_bkmrk_pml_quiesce_end(ompi_crcp_bkmrk_pml_quiesce_tag_type_t tag );
|
||||
|
||||
/*
|
||||
* Request function
|
||||
*/
|
||||
|
@ -61,6 +61,23 @@ typedef int (*ompi_crcp_base_module_init_fn_t)
|
||||
typedef int (*ompi_crcp_base_module_finalize_fn_t)
|
||||
(void);
|
||||
|
||||
|
||||
/************************
|
||||
* MPI Quiesce Interface
|
||||
************************/
|
||||
/**
|
||||
* MPI_Quiesce_start component interface
|
||||
*/
|
||||
typedef int (*ompi_crcp_base_quiesce_start_fn_t)
|
||||
(MPI_Info *info);
|
||||
|
||||
/**
|
||||
* MPI_Quiesce_end component interface
|
||||
*/
|
||||
typedef int (*ompi_crcp_base_quiesce_end_fn_t)
|
||||
(MPI_Info *info);
|
||||
|
||||
|
||||
/************************
|
||||
* PML Wrapper hooks
|
||||
* PML Wrapper is the CRCPW PML component
|
||||
@ -283,6 +300,10 @@ struct ompi_crcp_base_module_1_0_0_t {
|
||||
/** Finalization Function */
|
||||
ompi_crcp_base_module_finalize_fn_t crcp_finalize;
|
||||
|
||||
/**< MPI_Quiesce Interface Functions ******************/
|
||||
ompi_crcp_base_quiesce_start_fn_t quiesce_start;
|
||||
ompi_crcp_base_quiesce_end_fn_t quiesce_end;
|
||||
|
||||
/**< PML Wrapper Functions ****************************/
|
||||
ompi_crcp_base_pml_enable_fn_t pml_enable;
|
||||
|
||||
|
@ -32,6 +32,7 @@
|
||||
#include "orte/util/proc_info.h"
|
||||
|
||||
#if OPAL_ENABLE_FT_CR == 1
|
||||
#include "orte/mca/sstore/sstore.h"
|
||||
#include "ompi/mca/mpool/base/base.h"
|
||||
#include "ompi/runtime/ompi_cr.h"
|
||||
#endif
|
||||
@ -169,12 +170,12 @@ int mca_mpool_sm_ft_event(int state) {
|
||||
asprintf( &file_name, "%s"OPAL_PATH_SEP"shared_mem_pool.%s",
|
||||
orte_process_info.job_session_dir,
|
||||
orte_process_info.nodename );
|
||||
opal_crs_base_metadata_write_token(NULL, CRS_METADATA_TOUCH, file_name);
|
||||
orte_sstore.set_attr(orte_sstore_handle_current, SSTORE_METADATA_LOCAL_TOUCH, file_name);
|
||||
free(file_name);
|
||||
file_name = NULL;
|
||||
}
|
||||
else if(OPAL_CRS_CONTINUE == state) {
|
||||
if(ompi_cr_continue_like_restart) {
|
||||
if(orte_cr_continue_like_restart) {
|
||||
/* Find the sm module */
|
||||
self_module = mca_mpool_base_module_lookup("sm");
|
||||
self_sm_module = (mca_mpool_sm_module_t*) self_module;
|
||||
|
@ -691,7 +691,7 @@ int mca_pml_bfo_ft_event( int state )
|
||||
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P2);
|
||||
}
|
||||
|
||||
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
|
||||
if( orte_cr_continue_like_restart && !first_continue_pass ) {
|
||||
/*
|
||||
* Get a list of processes
|
||||
*/
|
||||
@ -791,7 +791,7 @@ int mca_pml_bfo_ft_event( int state )
|
||||
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P3);
|
||||
}
|
||||
|
||||
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
|
||||
if( orte_cr_continue_like_restart && !first_continue_pass ) {
|
||||
/*
|
||||
* Exchange the modex information once again.
|
||||
* BTLs will have republished their modex information.
|
||||
|
@ -669,7 +669,7 @@ int mca_pml_csum_ft_event( int state )
|
||||
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P2);
|
||||
}
|
||||
|
||||
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
|
||||
if( orte_cr_continue_like_restart && !first_continue_pass ) {
|
||||
/*
|
||||
* Get a list of processes
|
||||
*/
|
||||
@ -769,7 +769,7 @@ int mca_pml_csum_ft_event( int state )
|
||||
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P3);
|
||||
}
|
||||
|
||||
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
|
||||
if( orte_cr_continue_like_restart && !first_continue_pass ) {
|
||||
/*
|
||||
* Exchange the modex information once again.
|
||||
* BTLs will have republished their modex information.
|
||||
|
@ -638,7 +638,7 @@ int mca_pml_ob1_ft_event( int state )
|
||||
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P2);
|
||||
}
|
||||
|
||||
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
|
||||
if( orte_cr_continue_like_restart && !first_continue_pass ) {
|
||||
/*
|
||||
* Get a list of processes
|
||||
*/
|
||||
@ -738,7 +738,7 @@ int mca_pml_ob1_ft_event( int state )
|
||||
OPAL_CR_SET_TIMER(OPAL_CR_TIMER_P2P3);
|
||||
}
|
||||
|
||||
if( ompi_cr_continue_like_restart && !first_continue_pass ) {
|
||||
if( orte_cr_continue_like_restart && !first_continue_pass ) {
|
||||
/*
|
||||
* Exchange the modex information once again.
|
||||
* BTLs will have republished their modex information.
|
||||
|
38
ompi/mpiext/cr/Makefile.am
Normal file
38
ompi/mpiext/cr/Makefile.am
Normal file
@ -0,0 +1,38 @@
|
||||
#
|
||||
# Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
# University Research and Technology
|
||||
# Corporation. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
#
|
||||
# $HEADER$
|
||||
#
|
||||
|
||||
headers = \
|
||||
mpiext_cr_c.h
|
||||
|
||||
sources = \
|
||||
c/checkpoint.c \
|
||||
c/restart.c \
|
||||
c/migrate.c \
|
||||
c/inc_register_callback.c \
|
||||
c/quiesce_start.c \
|
||||
c/quiesce_end.c \
|
||||
c/quiesce_checkpoint.c \
|
||||
c/self_register_checkpoint.c \
|
||||
c/self_register_restart.c \
|
||||
c/self_register_continue.c
|
||||
|
||||
lib = libext_mpiext_cr.la
|
||||
lib_sources = $(sources)
|
||||
|
||||
extcomponentdir = $(pkglibdir)
|
||||
|
||||
noinst_LTLIBRARIES = $(lib)
|
||||
libext_mpiext_cr_la_SOURCES = $(lib_sources)
|
||||
libext_mpiext_cr_la_LDFLAGS = -module -avoid-version
|
||||
|
||||
ompidir = $(includedir)/openmpi/ompi/mpiext/cr
|
||||
ompi_HEADERS = \
|
||||
$(headers)
|
88
ompi/mpiext/cr/c/checkpoint.c
Normal file
88
ompi/mpiext/cr/c/checkpoint.c
Normal file
@ -0,0 +1,88 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
#include "ompi_config.h"
|
||||
#include <stdio.h>
|
||||
|
||||
#include "ompi/mpi/c/bindings.h"
|
||||
#include "ompi/info/info.h"
|
||||
#include "ompi/runtime/params.h"
|
||||
#include "ompi/communicator/communicator.h"
|
||||
#include "orte/mca/snapc/snapc.h"
|
||||
|
||||
#include "ompi/mpiext/cr/mpiext_cr_c.h"
|
||||
|
||||
static const char FUNC_NAME[] = "OMPI_CR_Checkpoint";
|
||||
#define HANDLE_SIZE_MAX 256
|
||||
|
||||
int OMPI_CR_Checkpoint(char **handle, int *seq, MPI_Info *info)
|
||||
{
|
||||
int ret = MPI_SUCCESS;
|
||||
MPI_Comm comm = MPI_COMM_WORLD;
|
||||
orte_snapc_base_request_op_t *datum = NULL;
|
||||
int state = 0;
|
||||
int my_rank;
|
||||
|
||||
/* argument checking */
|
||||
if (MPI_PARAM_CHECK) {
|
||||
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
|
||||
}
|
||||
|
||||
/*
|
||||
* Setup the data structure for the operation
|
||||
*/
|
||||
datum = OBJ_NEW(orte_snapc_base_request_op_t);
|
||||
datum->event = ORTE_SNAPC_OP_CHECKPOINT;
|
||||
datum->is_active = true;
|
||||
|
||||
MPI_Comm_rank(comm, &my_rank);
|
||||
if( 0 == my_rank ) {
|
||||
datum->leader = ORTE_PROC_MY_NAME->vpid;
|
||||
} else {
|
||||
datum->leader = -1; /* Unknown from non-root ranks */
|
||||
}
|
||||
|
||||
/*
|
||||
* All processes must make this call before it can start
|
||||
*/
|
||||
MPI_Barrier(comm);
|
||||
|
||||
/*
|
||||
* Leader sends the request
|
||||
*/
|
||||
OPAL_CR_ENTER_LIBRARY();
|
||||
ret = orte_snapc.request_op(datum);
|
||||
if( OMPI_SUCCESS != ret ) {
|
||||
OBJ_RELEASE(datum);
|
||||
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
|
||||
FUNC_NAME);
|
||||
}
|
||||
OPAL_CR_EXIT_LIBRARY();
|
||||
|
||||
/*
|
||||
* Leader then sends out the commit message
|
||||
*/
|
||||
if( datum->leader == (int)ORTE_PROC_MY_NAME->vpid ) {
|
||||
*handle = strdup(datum->global_handle);
|
||||
*seq = datum->seq_num;
|
||||
state = 0;
|
||||
} else {
|
||||
*handle = (char*)malloc(sizeof(char)*HANDLE_SIZE_MAX);
|
||||
}
|
||||
|
||||
MPI_Bcast(&state, 1, MPI_INT, 0, comm);
|
||||
MPI_Bcast(seq, 1, MPI_INT, 0, comm);
|
||||
MPI_Bcast(*handle, HANDLE_SIZE_MAX, MPI_CHAR, 0, comm);
|
||||
|
||||
datum->is_active = false;
|
||||
OBJ_RELEASE(datum);
|
||||
|
||||
return ret;
|
||||
}
|
39
ompi/mpiext/cr/c/inc_register_callback.c
Normal file
39
ompi/mpiext/cr/c/inc_register_callback.c
Normal file
@ -0,0 +1,39 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
#include "ompi_config.h"
|
||||
#include <stdio.h>
|
||||
|
||||
#include "ompi/mpi/c/bindings.h"
|
||||
#include "opal/runtime/opal_cr.h"
|
||||
#include "ompi/mpiext/cr/mpiext_cr_c.h"
|
||||
|
||||
#include "ompi/runtime/params.h"
|
||||
#include "ompi/communicator/communicator.h"
|
||||
#include "ompi/errhandler/errhandler.h"
|
||||
|
||||
static const char FUNC_NAME[] = "OMPI_CR_INC_register_callback";
|
||||
|
||||
int OMPI_CR_INC_register_callback(OMPI_CR_INC_callback_event_t event,
|
||||
OMPI_CR_INC_callback_function function,
|
||||
OMPI_CR_INC_callback_function *prev_function)
|
||||
{
|
||||
int rc;
|
||||
|
||||
if ( MPI_PARAM_CHECK ) {
|
||||
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
|
||||
}
|
||||
|
||||
OPAL_CR_ENTER_LIBRARY();
|
||||
|
||||
rc = opal_cr_user_inc_register_callback(event, function, prev_function);
|
||||
|
||||
OMPI_ERRHANDLER_RETURN(rc, MPI_COMM_WORLD, rc, FUNC_NAME);
|
||||
}
|
120
ompi/mpiext/cr/c/migrate.c
Normal file
120
ompi/mpiext/cr/c/migrate.c
Normal file
@ -0,0 +1,120 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
#include "ompi_config.h"
|
||||
#include <stdio.h>
|
||||
|
||||
#include "ompi/mpi/c/bindings.h"
|
||||
#include "ompi/info/info.h"
|
||||
#include "ompi/runtime/params.h"
|
||||
#include "ompi/communicator/communicator.h"
|
||||
#include "orte/mca/snapc/snapc.h"
|
||||
|
||||
#include "ompi/mpiext/cr/mpiext_cr_c.h"
|
||||
|
||||
static const char FUNC_NAME[] = "OMPI_CR_Migrate";
|
||||
|
||||
int OMPI_CR_Migrate(MPI_Comm comm, char *hostname, int rank, MPI_Info *info)
|
||||
{
|
||||
int ret = MPI_SUCCESS;
|
||||
orte_snapc_base_request_op_t *datum = NULL;
|
||||
int my_rank, my_size, i;
|
||||
char loc_hostname[MPI_MAX_PROCESSOR_NAME];
|
||||
int my_vpid;
|
||||
int info_flag;
|
||||
char info_value[6];
|
||||
int my_off_node = (int)false;
|
||||
|
||||
/* argument checking */
|
||||
if (MPI_PARAM_CHECK) {
|
||||
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
|
||||
}
|
||||
|
||||
/*
|
||||
* Setup the data structure for the operation
|
||||
*/
|
||||
datum = OBJ_NEW(orte_snapc_base_request_op_t);
|
||||
datum->event = ORTE_SNAPC_OP_MIGRATE;
|
||||
datum->is_active = true;
|
||||
|
||||
MPI_Comm_rank(comm, &my_rank);
|
||||
MPI_Comm_size(comm, &my_size);
|
||||
if( 0 == my_rank ) {
|
||||
datum->leader = ORTE_PROC_MY_NAME->vpid;
|
||||
} else {
|
||||
datum->leader = -1; /* Unknown from non-root ranks */
|
||||
}
|
||||
|
||||
/*
|
||||
* Gather all preferences to the root
|
||||
*/
|
||||
if( NULL == hostname ) {
|
||||
loc_hostname[0] = '\0';
|
||||
} else {
|
||||
strncpy(loc_hostname, hostname, strlen(hostname));
|
||||
loc_hostname[strlen(hostname)] = '\0';
|
||||
}
|
||||
my_vpid = (int) ORTE_PROC_MY_NAME->vpid;
|
||||
|
||||
if( 0 == my_rank ) {
|
||||
datum->mig_num = my_size;
|
||||
datum->mig_vpids = malloc(sizeof(int) * my_size);
|
||||
datum->mig_host_pref = malloc(sizeof(char) * my_size * MPI_MAX_PROCESSOR_NAME);
|
||||
datum->mig_vpid_pref = malloc(sizeof(int) * my_size);
|
||||
datum->mig_off_node = malloc(sizeof(int) * my_size);
|
||||
|
||||
for( i = 0; i < my_size; ++i ) {
|
||||
(datum->mig_vpids)[i] = 0;
|
||||
(datum->mig_host_pref)[i][0] = '\0';
|
||||
(datum->mig_vpid_pref)[i] = 0;
|
||||
(datum->mig_off_node)[i] = (int)false;
|
||||
}
|
||||
}
|
||||
|
||||
my_off_node = (int)false;
|
||||
if( NULL != info ) {
|
||||
MPI_Info_get(*info, "CR_OFF_NODE", 5, info_value, &info_flag);
|
||||
if( info_flag ) {
|
||||
if( 0 == strncmp(info_value, "true", strlen("true")) ) {
|
||||
my_off_node = (int)true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
MPI_Gather(&my_vpid, 1, MPI_INT,
|
||||
(datum->mig_vpids), 1, MPI_INT, 0, comm);
|
||||
MPI_Gather(loc_hostname, MPI_MAX_PROCESSOR_NAME, MPI_CHAR,
|
||||
(datum->mig_host_pref), MPI_MAX_PROCESSOR_NAME, MPI_CHAR, 0, comm);
|
||||
MPI_Gather(&my_vpid, 1, MPI_INT,
|
||||
(datum->mig_vpid_pref), 1, MPI_INT, 0, comm);
|
||||
MPI_Gather(&my_off_node, 1, MPI_INT,
|
||||
(datum->mig_off_node), 1, MPI_INT, 0, comm);
|
||||
|
||||
/*
|
||||
* Leader sends the request
|
||||
*/
|
||||
OPAL_CR_ENTER_LIBRARY();
|
||||
ret = orte_snapc.request_op(datum);
|
||||
if( OMPI_SUCCESS != ret ) {
|
||||
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
|
||||
FUNC_NAME);
|
||||
}
|
||||
OPAL_CR_EXIT_LIBRARY();
|
||||
|
||||
datum->is_active = false;
|
||||
OBJ_RELEASE(datum);
|
||||
|
||||
/*
|
||||
* All processes must sync before leaving
|
||||
*/
|
||||
MPI_Barrier(comm);
|
||||
|
||||
return ret;
|
||||
}
|
69
ompi/mpiext/cr/c/quiesce_checkpoint.c
Normal file
69
ompi/mpiext/cr/c/quiesce_checkpoint.c
Normal file
@ -0,0 +1,69 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
#include "ompi_config.h"
|
||||
#include <stdio.h>
|
||||
|
||||
#include "ompi/mpi/c/bindings.h"
|
||||
#include "ompi/info/info.h"
|
||||
#include "ompi/runtime/params.h"
|
||||
#include "ompi/communicator/communicator.h"
|
||||
#include "orte/mca/snapc/snapc.h"
|
||||
|
||||
#include "ompi/mpiext/cr/mpiext_cr_c.h"
|
||||
|
||||
static const char FUNC_NAME[] = "OMPI_CR_Quiesce_checkpoint";
|
||||
|
||||
int OMPI_CR_Quiesce_checkpoint(MPI_Comm commP, char **handle, int *seq, MPI_Info *info)
|
||||
{
|
||||
int ret = MPI_SUCCESS;
|
||||
MPI_Comm comm = MPI_COMM_WORLD; /* Currently ignore provided comm */
|
||||
orte_snapc_base_request_op_t *datum = NULL;
|
||||
int my_rank;
|
||||
|
||||
/* argument checking */
|
||||
if (MPI_PARAM_CHECK) {
|
||||
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
|
||||
}
|
||||
|
||||
/*
|
||||
* Setup the data structure for the operation
|
||||
*/
|
||||
datum = OBJ_NEW(orte_snapc_base_request_op_t);
|
||||
datum->event = ORTE_SNAPC_OP_QUIESCE_CHECKPOINT;
|
||||
datum->is_active = true;
|
||||
|
||||
MPI_Comm_rank(comm, &my_rank);
|
||||
if( 0 == my_rank ) {
|
||||
datum->leader = ORTE_PROC_MY_NAME->vpid;
|
||||
} else {
|
||||
datum->leader = -1; /* Unknown from non-root ranks */
|
||||
}
|
||||
|
||||
/*
|
||||
* Since we are quiescent, then this is a local operation
|
||||
*/
|
||||
OPAL_CR_ENTER_LIBRARY();
|
||||
ret = orte_snapc.request_op(datum);
|
||||
/*ret = ompi_crcp_base_quiesce_start(info);*/
|
||||
if( OMPI_SUCCESS != ret ) {
|
||||
OMPI_ERRHANDLER_INVOKE(MPI_COMM_WORLD, MPI_ERR_OTHER,
|
||||
FUNC_NAME);
|
||||
}
|
||||
OPAL_CR_EXIT_LIBRARY();
|
||||
|
||||
*handle = strdup(datum->global_handle);
|
||||
*seq = datum->seq_num;
|
||||
|
||||
datum->is_active = false;
|
||||
OBJ_RELEASE(datum);
|
||||
|
||||
return ret;
|
||||
}
|
74
ompi/mpiext/cr/c/quiesce_end.c
Normal file
74
ompi/mpiext/cr/c/quiesce_end.c
Normal file
@ -0,0 +1,74 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
#include "ompi_config.h"
|
||||
#include <stdio.h>
|
||||
|
||||
#include "ompi/mpi/c/bindings.h"
|
||||
#include "ompi/info/info.h"
|
||||
#include "ompi/runtime/params.h"
|
||||
#include "ompi/communicator/communicator.h"
|
||||
#include "orte/mca/snapc/snapc.h"
|
||||
|
||||
#include "ompi/mpiext/cr/mpiext_cr_c.h"
|
||||
|
||||
static const char FUNC_NAME[] = "OMPI_CR_Quiesce_end";
|
||||
|
||||
int OMPI_CR_Quiesce_end(MPI_Comm commP, MPI_Info *info)
|
||||
{
|
||||
int ret = MPI_SUCCESS;
|
||||
MPI_Comm comm = MPI_COMM_WORLD; /* Currently ignore provided comm */
|
||||
orte_snapc_base_request_op_t *datum = NULL;
|
||||
int my_rank;
|
||||
|
||||
/* argument checking */
|
||||
if (MPI_PARAM_CHECK) {
|
||||
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
|
||||
}
|
||||
|
||||
/*
|
||||
* Setup the data structure for the operation
|
||||
*/
|
||||
datum = OBJ_NEW(orte_snapc_base_request_op_t);
|
||||
datum->event = ORTE_SNAPC_OP_QUIESCE_END;
|
||||
datum->is_active = true;
|
||||
|
||||
MPI_Comm_rank(comm, &my_rank);
|
||||
if( 0 == my_rank ) {
|
||||
datum->leader = ORTE_PROC_MY_NAME->vpid;
|
||||
} else {
|
||||
datum->leader = -1; /* Unknown from non-root ranks */
|
||||
}
|
||||
|
||||
/*
|
||||
* Leader sends the request
|
||||
*/
|
||||
OPAL_CR_ENTER_LIBRARY();
|
||||
ret = orte_snapc.request_op(datum);
|
||||
/*ret = ompi_crcp_base_quiesce_end(info);*/
|
||||
if( OMPI_SUCCESS != ret ) {
|
||||
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
|
||||
FUNC_NAME);
|
||||
}
|
||||
OPAL_CR_EXIT_LIBRARY();
|
||||
|
||||
/*
|
||||
* All processes must make this call before it can complete
|
||||
*/
|
||||
MPI_Barrier(comm);
|
||||
|
||||
/*
|
||||
* (Old) info logic
|
||||
*/
|
||||
/*cur_datum.epoch = -1;*/
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
210
ompi/mpiext/cr/c/quiesce_start.c
Normal file
210
ompi/mpiext/cr/c/quiesce_start.c
Normal file
@ -0,0 +1,210 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
#include "ompi_config.h"
|
||||
#include <stdio.h>
|
||||
|
||||
#include "ompi/mpi/c/bindings.h"
|
||||
#include "ompi/info/info.h"
|
||||
#include "ompi/runtime/params.h"
|
||||
#include "ompi/communicator/communicator.h"
|
||||
#include "orte/mca/snapc/snapc.h"
|
||||
|
||||
#include "ompi/mpiext/cr/mpiext_cr_c.h"
|
||||
|
||||
static const char FUNC_NAME[] = "OMPI_CR_Quiesce_start";
|
||||
|
||||
int OMPI_CR_Quiesce_start(MPI_Comm commP, MPI_Info *info)
|
||||
{
|
||||
int ret = MPI_SUCCESS;
|
||||
MPI_Comm comm = MPI_COMM_WORLD; /* Currently ignore provided comm */
|
||||
orte_snapc_base_request_op_t *datum = NULL;
|
||||
int my_rank;
|
||||
|
||||
/* argument checking */
|
||||
if (MPI_PARAM_CHECK) {
|
||||
OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
|
||||
}
|
||||
|
||||
/*
|
||||
* Setup the data structure for the operation
|
||||
*/
|
||||
datum = OBJ_NEW(orte_snapc_base_request_op_t);
|
||||
datum->event = ORTE_SNAPC_OP_QUIESCE_START;
|
||||
datum->is_active = true;
|
||||
|
||||
MPI_Comm_rank(comm, &my_rank);
|
||||
if( 0 == my_rank ) {
|
||||
datum->leader = ORTE_PROC_MY_NAME->vpid;
|
||||
} else {
|
||||
datum->leader = -1; /* Unknown from non-root ranks */
|
||||
}
|
||||
|
||||
/*
|
||||
* All processes must make this call before it can start
|
||||
*/
|
||||
MPI_Barrier(comm);
|
||||
|
||||
/*
|
||||
* Leader sends the request
|
||||
*/
|
||||
OPAL_CR_ENTER_LIBRARY();
|
||||
ret = orte_snapc.request_op(datum);
|
||||
/*ret = ompi_crcp_base_quiesce_start(info);*/
|
||||
if( OMPI_SUCCESS != ret ) {
|
||||
OBJ_RELEASE(datum);
|
||||
OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_OTHER,
|
||||
FUNC_NAME);
|
||||
}
|
||||
|
||||
OPAL_CR_EXIT_LIBRARY();
|
||||
|
||||
datum->is_active = false;
|
||||
OBJ_RELEASE(datum);
|
||||
|
||||
/*
|
||||
* (Old) info logic
|
||||
*/
|
||||
/*ompi_info_set((ompi_info_t*)*info, "target", cur_datum.target_dir);*/
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
/*****************
|
||||
* Local Functions
|
||||
******************/
|
||||
#if 0
|
||||
/* Info keys:
|
||||
*
|
||||
* - crs:
|
||||
* none = (Default) No CRS Service
|
||||
* default = Whatever CRS service MPI chooses
|
||||
* blcr = BLCR
|
||||
* self = app level callbacks
|
||||
*
|
||||
* - cmdline:
|
||||
* Command line to restart the process with.
|
||||
* If empty, the user must manually enter it
|
||||
*
|
||||
* - target:
|
||||
* Absolute path to the target directory.
|
||||
*
|
||||
* - handle:
|
||||
* first = Earliest checkpoint directory available
|
||||
* last = Most recent checkpoint directory available
|
||||
* [global:local] = handle provided by the MPI library
|
||||
*
|
||||
* - restarting:
|
||||
* 0 = not restarting
|
||||
* 1 = restarting
|
||||
*
|
||||
* - checkpointing:
|
||||
* 0 = No need to prepare for checkpointing
|
||||
* 1 = MPI should prepare for checkpointing
|
||||
*
|
||||
* - inflight:
|
||||
* default = message
|
||||
* message = Drain inflight messages at the message level
|
||||
* network = Drain inflight messages at the network level (if possible)
|
||||
*
|
||||
* - user_space_mem:
|
||||
* 0 = Memory does not need to be managed
|
||||
* 1 = Memory must be in user space (i.e., not on network card
|
||||
*
|
||||
*/
|
||||
static int extract_info_into_datum(ompi_info_t *info, orte_snapc_base_quiesce_t *datum)
|
||||
{
|
||||
int info_flag = false;
|
||||
int max_crs_len = 32;
|
||||
bool info_bool = false;
|
||||
char *info_char = NULL;
|
||||
|
||||
info_char = (char *) malloc(sizeof(char) * (OPAL_PATH_MAX+1));
|
||||
|
||||
/*
|
||||
* Key: crs
|
||||
*/
|
||||
ompi_info_get(info, "crs", max_crs_len, info_char, &info_flag);
|
||||
if( info_flag) {
|
||||
datum->crs_name = strdup(info_char);
|
||||
}
|
||||
|
||||
/*
|
||||
* Key: cmdline
|
||||
*/
|
||||
ompi_info_get(info, "cmdline", OPAL_PATH_MAX, info_char, &info_flag);
|
||||
if( info_flag) {
|
||||
datum->cmdline = strdup(info_char);
|
||||
}
|
||||
|
||||
/*
|
||||
* Key: handle
|
||||
*/
|
||||
ompi_info_get(info, "handle", OPAL_PATH_MAX, info_char, &info_flag);
|
||||
if( info_flag) {
|
||||
datum->handle = strdup(info_char);
|
||||
}
|
||||
|
||||
/*
|
||||
* Key: target
|
||||
*/
|
||||
ompi_info_get(info, "target", OPAL_PATH_MAX, info_char, &info_flag);
|
||||
if( info_flag) {
|
||||
datum->target_dir = strdup(info_char);
|
||||
}
|
||||
|
||||
/*
|
||||
* Key: restarting
|
||||
*/
|
||||
ompi_info_get_bool(info, "restarting", &info_bool, &info_flag);
|
||||
if( info_flag ) {
|
||||
datum->restarting = info_bool;
|
||||
} else {
|
||||
datum->restarting = false;
|
||||
}
|
||||
|
||||
/*
|
||||
* Key: checkpointing
|
||||
*/
|
||||
ompi_info_get_bool(info, "checkpointing", &info_bool, &info_flag);
|
||||
if( info_flag ) {
|
||||
datum->checkpointing = info_bool;
|
||||
} else {
|
||||
datum->checkpointing = false;
|
||||
}
|
||||
|
||||
/*
|
||||
* Display all values
|
||||
*/
|
||||
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: %s extract_info: Info('crs' = '%s')",
|
||||
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
||||
(NULL == datum->crs_name ? "Default (none)" : datum->crs_name)));
|
||||
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: %s extract_info: Info('cmdline' = '%s')",
|
||||
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
||||
(NULL == datum->cmdline ? "Default ()" : datum->cmdline)));
|
||||
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: %s extract_info: Info('checkpointing' = '%c')",
|
||||
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
||||
(datum->checkpointing ? 'T' : 'F')));
|
||||
OPAL_OUTPUT_VERBOSE((3, mca_crcp_bkmrk_component.super.output_handle,
|
||||
"crcp:bkmrk: %s extract_info: Info('restarting' = '%c')",
|
||||
ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
|
||||