1
1
Граф коммитов

2759 Коммитов

Автор SHA1 Сообщение Дата
Rolf vandeVaart
cdd2d09c69 Fix broken compile.
This commit was SVN r23167.
2010-05-18 12:43:21 +00:00
Jeff Squyres
76d1bb3c96 Per some off-list discussion with Ralph and George, MPIR_being_debugged
really should be volatile.

This commit was SVN r23166.
2010-05-18 11:42:41 +00:00
Abhishek Kulkarni
260eaa24dd Add a missing header. Bleh.
This commit was SVN r23165.
2010-05-17 23:55:53 +00:00
Abhishek Kulkarni
8335ff3893 Fixing a branch merge issue. Looks like we picked the wrong branch when resolving
a conflict.

This commit was SVN r23164.
2010-05-17 23:49:46 +00:00
Abhishek Kulkarni
afbe3e99c6 * Wrap all the direct error-code checks of the form (OMPI_ERR_* == ret) with
(OMPI_ERR_* = OPAL_SOS_GET_ERR_CODE(ret)), since the return value could be a
 SOS-encoded error. The OPAL_SOS_GET_ERR_CODE() takes in a SOS error and returns
 back the native error code.

* Since OPAL_SUCCESS is preserved by SOS, also change all calls of the form
  (OPAL_ERROR == ret) to (OPAL_SUCCESS != ret). We thus avoid having to
  decode 'ret' to get the native error code.

This commit was SVN r23162.
2010-05-17 23:08:56 +00:00
Abhishek Kulkarni
f7f4dd87ab Change ORTE_ERR_LOG macro to make it SOS-aware.
If the logged error is an SOS-encoded error, use the SOS log function otherwise
use the existing errmgr log function.

This commit was SVN r23161.
2010-05-17 23:02:13 +00:00
Abhishek Kulkarni
9c5860706f Merge improvements to the "notifier" framework from the OPAL SOS and the ORTE WDC mercurial branches into the SVN trunk.
A brief description of the improvements can be found at
https://svn.open-mpi.org/trac/ompi/wiki/ORTEWDC#ChangesdonetotheORTEnotifier

This commit was SVN r23157.
2010-05-17 22:48:05 +00:00
Abhishek Kulkarni
f5b9bc4ff1 Add a new "HNP" component to the notifier framework.
This component proxies notification messages up to the HNP.  This
component runs in both the HNP and non-HNP processes for ease of
selection (e.g., so you can "--mca notifier hnp" (vs. "--mca
notifier hnp,non_hnp").  It auto-detects where it is running and
does the Right Thing -- if it's in the HNP process, it sets up to
receive incoming proxied messages.  If it's not in the HNP, then it
proxies all messages to the HNP.

This commit was SVN r23156.
2010-05-17 22:43:43 +00:00
Abhishek Kulkarni
197ec7586d Add a new "file" component to the notifier framework.
When this component is selected, the notification messages are sent to a file. 
The file can be a plain file or stdout or stderr.

The MCA parameter "notifier_file_name" can be used to specify the suffix of the file the notification messages should be sent to.
The default suffix is "wdc" and the file full name is "output-wdc".

This commit was SVN r23155.
2010-05-17 22:39:52 +00:00
Ralph Castain
7e6985edbf Cleanup warnings
This commit was SVN r23150.
2010-05-16 20:23:26 +00:00
Ralph Castain
de85049477 Cleanup warnings
This commit was SVN r23149.
2010-05-16 20:22:18 +00:00
Ralph Castain
88f5217a12 Cleanup the debugger daemon co-launch code and add an ability to test it. Implement ability to co-launch debugger daemons upon attach to a running job for jobs launched under rsh, slurm, and tm environments (others can easily be added if desired).
Add new mca params to test:

orte_debugger_test_daemon: Name of the executable to be used to simulate a debugger colaunch
orte_debugger_test_attach: Test debugger colaunch after debugger attachment

To test co-launch at job start, just set the orte_debugger_test_daemon param.

To test co-launch upon attach:
set orte_debugger_test_daemon
set orte_debugger_test_attach=1
set orte_enable_debug_cospawn_while_running=1
set orte_debugger_check_rate=<N> - defines the number of seconds to wait before "checking" for a debugger attaching

Added a "debugger" program to orte/test/mpi that just spins to simulate a debugger daemon.

This commit was SVN r23144.
2010-05-14 18:44:49 +00:00
Ralph Castain
b9f0615727 Correct some logic for tracking launch progress
This commit was SVN r23122.
2010-05-12 18:39:10 +00:00
Ralph Castain
7ce34223f1 Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases.
Implement process migration when failed nodes are detected. Some testing still required

This commit was SVN r23121.
2010-05-12 18:11:58 +00:00
Ralph Castain
306533fdb8 Replace a missing line that shutdown a peer that failed comm.
This commit was SVN r23120.
2010-05-12 18:09:35 +00:00
Ralph Castain
871f445848 Ignore nodes that are "down" when generating maps
This commit was SVN r23119.
2010-05-12 18:08:40 +00:00
Ralph Castain
4bd25f587c Begin handling the case of lost connections by having the OOB report it to the errmgr instead of the routed framework. Add an "app" component to t
he errmgr framework so that it can decide how to respond - which for now at least is just to check for lifeline and abort if so.

Add a new error constant to indicate that the error is "unrecoverable" so the oob can know it needs to abort.

This commit was SVN r23112.
2010-05-11 00:34:12 +00:00
Ralph Castain
86033074ce Grr...it's a job state, not a proc state
This commit was SVN r23110.
2010-05-07 16:50:10 +00:00
Ralph Castain
70bef5ca95 It is possible for the jdata param to be NULL, so protect the verbose output in that case
This commit was SVN r23109.
2010-05-07 16:48:44 +00:00
Shiqing Fan
8ea8563462 Include the new component config file for Windows into tarball.
This commit was SVN r23108.
2010-05-07 14:30:12 +00:00
Ralph Castain
d6a1d7a082 Little more cleanup on paffinity. Provide a specific error code for affinity not supported so we can better report the problem. Move the error reporting to orterun so we only get one error message. Update the darwin paffinity module to return the correct new error codes.
This commit was SVN r23107.
2010-05-07 14:04:55 +00:00
Ralph Castain
d4f56cff61 More cleanup on paffinity....groan
It is okay to not have a paffinity module IF you aren't using paffinity anyway. So don't error out of MPI_Init because a paffinity module wasn't selected.

Cleanup error reporting in the odls default module to (once and for all!) eliminate messages originating in the fork'd process. Create some new error codes to allow us to pass enough info back to the parent process to provide useful error messages.

This commit was SVN r23106.
2010-05-06 20:57:17 +00:00
Jeff Squyres
477201e161 Fix "make dist" breakage
This commit was SVN r23105.
2010-05-06 18:47:20 +00:00
Shiqing Fan
76726f6094 Update a few module configuration for Windows.
This commit was SVN r23104.
2010-05-05 12:22:04 +00:00
Ralph Castain
2ff1ae13e1 Create a new "heartbeat" module in the sensor framework and move the plm_base heartbeat code there. Add new proc and job states for heartbeat_failed. Remove the "heartbeat" cmd line option for orted as this is now done automatically if the --enable-heartbeat configure option is set.
This commit was SVN r23102.
2010-05-05 00:48:43 +00:00
Ralph Castain
99f223210d Add some contributed examples of how to start and configure the spread library. Do a little more cleanup on the spread module, and ensure that it isn't selected if spread isn't running.
This commit was SVN r23101.
2010-05-04 23:44:00 +00:00
Ralph Castain
43005b88e0 Catch one more spot...thanks to Sam for reporting it
This commit was SVN r23094.
2010-05-04 17:21:15 +00:00
Ralph Castain
f4ae2885e2 Add new error constant
This commit was SVN r23090.
2010-05-04 13:44:33 +00:00
Ralph Castain
cd569f8a79 Restore the global restart capability
This commit was SVN r23089.
2010-05-04 02:40:29 +00:00
Ralph Castain
3ca0b4138b Let the nidmap functions update a new orte_process_info field as to the number of daemons in the system
This commit was SVN r23088.
2010-05-04 02:40:09 +00:00
Ralph Castain
8c5f442ee0 Fix some bugs in the spread rmcast component
This commit was SVN r23086.
2010-05-04 02:38:37 +00:00
Ralph Castain
8e7faf9119 Add a new test for the db framework, fix some minor bugs in the daemon module
This commit was SVN r23085.
2010-05-04 02:38:11 +00:00
Ralph Castain
9dfb5c7c62 Rename the orte state framework to be "db", which more accurately reflects its overall capabilities since it can store any kind of data (not just state, although that will be its primary purpose). Update tools and tests accordingly. Add a daemon module for storing data on the daemons - requires --enable-multicast, so it won't build unless that is set
This commit was SVN r23082.
2010-05-03 04:11:03 +00:00
Ralph Castain
5103ffead6 Continue work on local recovery for orteds
This commit was SVN r23080.
2010-05-03 04:08:13 +00:00
Ralph Castain
bcff0d6301 Some minor cleanup in the rmcast framework, ensure that a default multicast group is always defined for each app
This commit was SVN r23079.
2010-05-03 04:07:14 +00:00
Ralph Castain
f994a7edf4 Add recovery data to the jobdat object
This commit was SVN r23078.
2010-05-03 04:06:13 +00:00
Ralph Castain
323224b84b Allow the restart data to be retrieved from a job's app->env or default that given to orterun
This commit was SVN r23077.
2010-05-03 04:05:19 +00:00
Ralph Castain
3f262bf0b6 Add a new reliable multicast component based on the "spread" library
Thanks to Srini Nariangadu (Cisco) for the contribution!

This commit was SVN r23076.
2010-05-02 17:29:41 +00:00
Ralph Castain
c93af95351 Since there is no defined behavior for the case where all application procs exit normally, but some or all have non-zero returns, just output a warning telling the user how many procs meet that criteria. Let the return code of mpirun in that scenario reflect any errors in OMPI/ORTE itself.
Clearly a temporary solution until a defined behavior can be established.

This commit was SVN r23075.
2010-04-30 19:01:10 +00:00
Ralph Castain
c62418d76d Add a new test that checks behavior when we call exit with a non-zero return code after calling finalize - don't ask why.
Modify the check_complete code so it finds the first non-zero exit status (i.e., the one from the lowest rank) in a job that terminates normally, and sets the mpirun exit code to that status.

This commit was SVN r23071.
2010-04-29 19:58:44 +00:00
Ralph Castain
5689bd6d30 Remove debug
This commit was SVN r23062.
2010-04-28 22:14:23 +00:00
Ralph Castain
319758e3e0 Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params:
1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis

2. orte_max_local_restarts - default max number of local restarts, can be overridden

3. orte_max_global_restarts - default max number of relocates, can be overridden

Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code.

This commit was SVN r23057.
2010-04-28 04:06:57 +00:00
Ralph Castain
67568e7dec Fix a warning
This commit was SVN r23049.
2010-04-27 15:59:24 +00:00
Ralph Castain
401fb22847 Fix the pack/unpack of process state values by getting the actual packing size to match the new value size.
Improve the debug output by printing the string

This commit was SVN r23048.
2010-04-27 15:58:04 +00:00
Ralph Castain
efc7d3e2f6 Update the LSF PLM module to ensure that the path is correctly passed.
Many thanks to Teng Lin for the error report - and the patch!

This commit was SVN r23047.
2010-04-27 03:46:04 +00:00
Ralph Castain
3434296836 Ensure we don't have a trailing separator on the end of our tmpdir as (a) it really looks weird, and (b) some exotic systems interpret that as indicating the rest of the path is to be treated as absolute. Makes for very strange and interesting behavior...
This commit was SVN r23046.
2010-04-27 03:40:44 +00:00
Ralph Castain
55889934d8 After hours spent chasing the stupid "abort" file, it became clear that we were always going to be plagued by that idiot contraption when trying to be good citizens and properly cleanup. So get rid of it by instead doing a messaging handshake with the local daemon.
Note that this isn't a problem since MPI_Abort and orte_abort are only called under controlled circumstances - i.e., we are doing an orderly abort and not segfaulting. If we can't get the message out for some reason, then too bad - we'll still see an abnormal process termination and act accordingly.

This commit was SVN r23045.
2010-04-27 03:39:32 +00:00
Ralph Castain
b9893aacc5 Add a sensor framework to ORTE that monitors applications and notifies the errmgr when they exceed specified boundaries. Two modules are included here:
1. file activity - can monitor file size, access and modification times. If these fail to change over a specified number of sampling iterations (rate is an mca param), then the errmgr is notified.

2. memory usage - checks amount of memory used by a process. Limit and sampling rate can be set.

This support must be enabled by configuring --enable-sensors.

ompi_info and orte-info have been updated to include the new framework.

Also includes some initial steps toward restoring the recovery capability. Most notably, the ODLS API has been extended to include a "restart_proc" entry for restarting a local process, and organizes the various ERRMGR framework globals into a single struct as we do in the other ORTE frameworks. Fix an oversight in the ERRMGR framework where a pointer array was constructed, but not initialized.

Implementation continues.

This commit was SVN r23043.
2010-04-26 22:15:57 +00:00
Ralph Castain
43a89bbace Extend the process and job states by adding values for exceeding sensor bounds. This changes the job state field to 32-bit to also provide room for future expansion.
This commit was SVN r23036.
2010-04-26 12:36:40 +00:00
Ralph Castain
e3164d2ac1 For the autogen-challenged (i.e., Jeff), create a new ORTE constant that tells the user that a required module was not found. Update the errmgr select function to output the error if no module is found.
This commit was SVN r23032.
2010-04-24 01:39:26 +00:00