1
1

2405 Коммитов

Автор SHA1 Сообщение Дата
Ralph Castain
a8cf19a7bc Ensure heartbeat only started once and only for daemon job
This commit was SVN r24416.
2011-02-18 20:33:54 +00:00
Ralph Castain
ef56e6d78b Helps to move the pointer
This commit was SVN r24414.
2011-02-18 14:01:25 +00:00
Ralph Castain
7b35ada7fc Fix ricochet effect - move failed procs to next on list instead of loadbalancing
This commit was SVN r24413.
2011-02-18 13:11:55 +00:00
Ralph Castain
b98a2917ff Add an API to the errmgr so that apps can register for a callback to warn them of an impending migration - this gives apps a chance to cleanly terminate prior to being migrated for external reasons (e.g., impending failures). The timeout provided indicates to the daemon how long it should wait before proceeding to kill/migrate the process - if the process fails to exit before that time, the daemon will kill it.
This commit was SVN r24412.
2011-02-18 02:48:12 +00:00
Ralph Castain
51cf0a16c3 Some minor cleanups to support VM and CM operations
This commit was SVN r24408.
2011-02-16 23:03:08 +00:00
Ralph Castain
9b48c07599 CM daemons handle their own output
This commit was SVN r24407.
2011-02-16 23:02:23 +00:00
Ralph Castain
65ba6af44d Cleanup our handling of VMs to ensure daemons don't get mapped when operating with a VM.
Have each mapper flag it did the map so we can see who did it later.

Ensure procs are flagged as "ready to launch".

This commit was SVN r24406.
2011-02-16 23:01:57 +00:00
Jeff Squyres
3f4d4886f2 Minor update for something that has been bugging me for quite a while:
OMPI supports multiple different repository systems (SVN, hg, git).
But the VERSION file has listed "want_svn" and "svn_r" as fields, even
though the actual repo system and version may not be SVN.

So search/replace those fields (and derrivative values that come from
those fields) with "want_repo_rev" and "repo_rev", respectively.

This commit was SVN r24405.
2011-02-16 22:53:23 +00:00
Ralph Castain
a32a7d9a82 Update heartbeat system
This commit was SVN r24404.
2011-02-16 18:50:51 +00:00
Ralph Castain
9b38525d1e Remove unused include files
This commit was SVN r24394.
2011-02-16 00:32:47 +00:00
Ralph Castain
5120e6aec3 Redefine the rmaps framework to allow multiple mapper modules to be active at the same time. This allows users to map the primary job one way, and map any comm_spawn'd job in a different way. Modules are given the opportunity to map a job in priority order, with the round-robin mapper having the highest default priority. Priority of each module can be defined using mca param.
When called, each mapper checks to see if it can map the job. If npernode is provided, for example, then the loadbalance mapper accepts the assignment and performs the operation - all mappers before it will "pass" as they can't map npernode requests.

Also remove the stale and never completed topo mapper.

This commit was SVN r24393.
2011-02-15 23:24:31 +00:00
Ralph Castain
c1da94a444 Dead children have no pid
This commit was SVN r24387.
2011-02-15 13:30:51 +00:00
Ralph Castain
a3607ff35d Make it easier to send a kill-local-procs command for an arbitrary number of procs
This commit was SVN r24386.
2011-02-15 13:26:11 +00:00
Ralph Castain
bf1cff3711 Plug a couple of additional memory leaks - try to highlight a little better that strings returned from reg_string_name must be freed by caller
This commit was SVN r24383.
2011-02-14 20:58:22 +00:00
Ralph Castain
a9dca25ca5 Remove the distinction between local and global restarts - leave it up to the error strategy to decide which to do.
Cleanup the heartbeat handling so it is associated with the proc, not a node.

Cleanup handling of recovery options so that defaults do not override user values iff they are provided.

This commit was SVN r24382.
2011-02-14 20:49:12 +00:00
Ralph Castain
b5de068533 Clean up an error in r24371 - can't use a const parameter as target in asprintf as it changes the value of the address.
Add some new proc/job states

Rename a constant to reflect coming change - remove the arbitrary difference between restarting a proc locally and relocating it to another node in terms of the number of restarts allowed.

Add pretty-print of signals for "proc aborted due to signal" reports.

This commit was SVN r24378.

The following SVN revision numbers were found above:
  r24371 --> open-mpi/ompi@93d28a5792
2011-02-14 19:29:09 +00:00
Abhishek Kulkarni
93d28a5792 Change opal_err2str_fn_t to return the error string as an argument.
This means that the converters (opal_err2str, orte_err2str) can now
return NULL as a "silent error". The return value of opal_err2str_fn_t
is the status of the operation (OPAL_SUCCESS or OPAL_ERROR).

This fixes the "Unknown error" message issues on the trunk.

This commit was SVN r24371.
2011-02-13 16:09:17 +00:00
Ralph Castain
33b68132cc Update the rmcast framework
This commit was SVN r24370.
2011-02-12 16:52:03 +00:00
Josh Hursey
a9335ea423 Make sure to initialize the 'update_state' function for the default module.
This will prevent tools from segfaulting if the mpirun process goes away suddenly while they are trying to communicate with it over the OOB.

This commit was SVN r24365.
2011-02-08 20:42:32 +00:00
Nysal Jan
3a8d251daa vsyslog is not included in SUSv3. Add a check for platforms that do not have vsyslog
This commit was SVN r24339.
2011-02-02 10:05:57 +00:00
Josh Hursey
fa3f6485d8 Make sure to define the region of time in which the migration is occurring so that the automatic recovery does not jump in the middle when we are moving processes around.
This commit was SVN r24326.
2011-01-31 19:09:47 +00:00
Josh Hursey
5b58ff0663 Fix a C/R checkpoint->restart->checkpoint->restart case.
The problem is that the SStore components were not flushing the old, stale checkpoint information. As a result the checkpoint was writing into the wrong directory, which produced an invalid checkpoint.

This seems to be fixed now. Thanks to Alex Brick for the bug report.

This commit was SVN r24325.
2011-01-28 21:25:14 +00:00
Josh Hursey
8ec85c6b8f Fixes the C/R Automatic Recovery feature when the HNP is also hosting processes locally.
I want to thank Hugo Meyer for reporting this/these bugs.

Notes:
 * Moved over a patch from the stabilization branch that makes sure we close the peer socket in the OOB TCP component fully during shutdown (after the de-registration sync). It also ensures that we free the rml_uri only after we are done communicating with the peer (in the odls_base deregister sync operation).
 * When an error is detected while delivering messages, we really want to bail out of the loop since the error manager is likely mutating the orte_local_children data structure, so it is no longer safe to iterate over in the orte_odls_base_default_deliver_message() function.
 * When the HNP is hosting processes make sure it accounts for processes that may have failed locally in the ErrMgr HNP component by decrementing the num_local_procs. This makes it match the orted ErrMgr component accounting. This is what was causing the modex to fail (the number of participants was wrong on a rolling recovery.
 * The crmig and autor features of the hnp ErrMgr component now check for the jobid from both the 'job' parameter and from the process name (since one may be there and not the other). This caused some additional error messages during startup.
 * If we fail to migrate (e.g., due to invalid node specification), print only the error message, not the error and success messages. This can be misleading.

This commit was SVN r24317.
2011-01-27 20:40:23 +00:00
Josh Hursey
81fd41f811 Return an informative error message if the user requests a migration of a job that is not capable of it.
C/R Functionality cleanup

This commit was SVN r24307.
2011-01-26 15:36:34 +00:00
Josh Hursey
8f45fcb429 More fixes for the C/R support. Fixes a couple bugs with the migration and autor features. The C/R functionality should be fully working now.
* Fix the checkpoint-restart-checkpoint case which would previous reject the checkpoint of the newly restarted process. By making sure to re-enable checkpointing once the application has fully restarted fixes this issue (make sure to set is_app_checkpointable to true on restart confirmation).
 * In the case of an invalid checkpoint, do not try to access the SStore datastore as it will be using a dummy handler, and return NULL strings. mpirun was segfaulting in the error case because it was trying to convert the seq_num from a string to an integer.
 * Make sure to initialize the timer event in the Automatic Recovery section of the HNP errmgr, per the libevent update. This caused a segfault when attempting to recover a failed process.
 * If ompi-checkpoint loses connection to the HNP/mpirun the TCP socket will fail and call the ErrMgr update_state function. This commit adds a dummy function {{{orte_errmgr_base_update_state()}}} that will prevent the ompi-checkpoint command from segfaulting in this error scenario.

This commit was SVN r24306.
2011-01-26 14:56:35 +00:00
Nathan Hjelm
8a3179cdcb removed c99 test code
This commit was SVN r24297.
2011-01-25 23:02:35 +00:00
Nathan Hjelm
e2126512a9 test c99 struct initialization with mtt. remove on jan 20, 2011
This commit was SVN r24271.
2011-01-19 22:21:21 +00:00
Abhishek Kulkarni
fd7ef7a1f1 Fixes broken trunk compile: call process status notify
only when ft-enable-cr is selected.

This commit was SVN r24255.
2011-01-14 18:37:07 +00:00
Abhishek Kulkarni
87d2c9b31d Few fault tolerance updates related to the CIFTS project (http://www.mcs.anl.gov/research/cifts/)
* Improve the FTB notifier to publish (C/R, process/communication failure) events to the FTB with the
   OMPI jobid as the associated payload.
 * Add notifier calls for C/R events and process status events in SnapC and ErrMgr components.
 * Fix a bug where the SnapC states and process states collide before being thrown out over the notifier.

This commit was SVN r24251.
2011-01-13 20:13:49 +00:00
Ralph Castain
b09f57b03d Update the multicast subsystem - ported from Cisco branch
This commit was SVN r24246.
2011-01-13 01:54:05 +00:00
Terry Dontje
f3aaa885a3 corrected a couple places in orte where it said cpu_model when it should have been cpu_type.
This commit was SVN r24221.
2011-01-11 19:56:26 +00:00
Abhishek Kulkarni
11ffa854ff Update the FTB notifier
* fix indentation issues
 * update the name of one of the fault events published to the FTB (per the FTB MPI standard)

This commit was SVN r24213.
2011-01-10 18:58:31 +00:00
Nathan Hjelm
c082d05ecb Reset the timer on MPIR_being_debugged only if MPIR_being_debugged is not set. Fix typo in return code.
This commit was SVN r24187.
2010-12-20 21:00:49 +00:00
Ralph Castain
2dc5cbb483 Remove stale code and API from the RML/OOB frameworks. Stopped using this code years ago.
This commit was SVN r24153.
2010-12-05 15:58:21 +00:00
Rolf vandeVaart
b67d3398da It is convention to have orte_config.h included at top of file.
This commit was SVN r24146.
2010-12-03 16:13:31 +00:00
Shiqing Fan
f43862420c Convert the bad dos line endings to unix style for all windows related files.
This commit was SVN r24137.
2010-12-02 12:08:08 +00:00
Ralph Castain
aaad8ae891 Remove unused var
This commit was SVN r24136.
2010-12-02 02:38:13 +00:00
Ralph Castain
f9ffff59f8 Ensure clean termination of threads and tcp multicast
This commit was SVN r24134.
2010-12-02 00:23:42 +00:00
Nathan Hjelm
75605faa75 added support for reattaching a debugger using the MPIR_attach_fifo
This commit was SVN r24132.
2010-12-01 20:13:58 +00:00
Ralph Castain
ad814f26cd One more time, into the breach!
Restore the use of override_oversubscribe to indicate that the data source for resources on the backend nodes used in mapping is unreliable. In this situation (e.g., data came from hostfile, or we are just using localhost because nothing was provided), we don't trust the oversubscribe condition passed by the mapper. Instead, we check locally to ensure we set sched_yield correctly.

This commit was SVN r24130.
2010-12-01 15:15:26 +00:00
Ralph Castain
eba65e97f3 Extend the rmcast APIs to allow enable/disable of comm, required for clean termination by upper layer users.
Point the recv thread event base to the right place so it can wakeup when required.

Add a new error code for "comm disabled" when attempting to communicate after disabling comm.

This commit was SVN r24129.
2010-12-01 13:41:19 +00:00
Ralph Castain
9224302c10 Remove debug
This commit was SVN r24128.
2010-12-01 13:12:24 +00:00
Ralph Castain
30c37ea536 Ensure that the oversubscribed condition of nodes is accurately reported by the mapper, and that the results are communicated and used by the backend orteds when setting sched_yield on local procs. Restores prior behavior that was somehow lost along the way.
Includes a patch from Damien Guinier to fix vpid assignments when cpus-per-task is specified.

This commit was SVN r24126.
2010-12-01 12:51:39 +00:00
Ralph Castain
85a974b0de Better check for NULL before using the value
This commit was SVN r24122.
2010-12-01 04:48:50 +00:00
Ralph Castain
c56185887b Change the event base "wakeup" support to enable the passing of events to the central thread for add/del. Add a macro OPAL_UPDATE_EVBASE for this purpose as it will likely be widely used.
Update the ORTE thread support to utilize this capability. Update the rmcast framework to track the change.

This commit was SVN r24121.
2010-12-01 04:26:43 +00:00
Ralph Castain
0441e81882 Oops - ensure that multicast msgs get circulated properly with the tcp module
This commit was SVN r24118.
2010-11-30 21:13:53 +00:00
Ralph Castain
d20c023348 Checkpoint the threading support for multicast - will be revised shortly, but this version currently works.
This commit was SVN r24117.
2010-11-30 17:30:16 +00:00
Ralph Castain
71669720a3 Just get the output once on sigpipe error, and include the fd
This commit was SVN r24092.
2010-11-25 15:32:48 +00:00
Ralph Castain
30c635fd4d Don't endlessly output sigpipe errors. Count the number of times we trap it, and abort if we get more than 10 of them.
This commit was SVN r24091.
2010-11-25 15:25:24 +00:00
Rolf vandeVaart
09fdd5cc23 Include fcntl.h, not sys/fcntl.h so we get the definition
of the open system call.  That is what man page says to do.
Fixes warning on Solaris.

This commit was SVN r24073.
2010-11-19 17:40:02 +00:00