1
1
Граф коммитов

2225 Коммитов

Автор SHA1 Сообщение Дата
Ralph Castain
cfbfbb75a2 Don't abort if a race condition causes a nid not to be found
This commit was SVN r23513.
2010-07-27 16:20:58 +00:00
Ralph Castain
06fe2c4c20 Add debug
This commit was SVN r23512.
2010-07-27 16:20:29 +00:00
Ralph Castain
e16ec40283 Check the correct envar. Thanks to Philippe for pointing out the error!
This commit was SVN r23499.
2010-07-27 03:49:03 +00:00
Ralph Castain
0c201486e4 Allow for multiple channels for the same channel "name" - could be both input and output channels
This commit was SVN r23498.
2010-07-27 01:38:39 +00:00
Ralph Castain
6158cab2e1 Add missing header
This commit was SVN r23497.
2010-07-27 01:37:41 +00:00
Ralph Castain
1ba8bbe1a9 Don't require the existence of a multicast interface if --enable-multicast wasn't specified.
This commit was SVN r23494.
2010-07-26 15:09:57 +00:00
Ralph Castain
e858b029aa Re-establish support for IBM's POE environment by resurrecting the poe launcher from the v1.2 series, updated for the current trunk. Modify the hnp errmgr module to support batch-operated jobs. Update the new ess generic module to support a wider range of mapping options.
This commit was SVN r23493.
2010-07-24 23:35:20 +00:00
Ralph Castain
ff2d573f7e Adjust some default values, and ensure we don't start sending too soon
This commit was SVN r23492.
2010-07-23 19:37:16 +00:00
Ralph Castain
140e427a79 Ensure that wildcard recvs go to the end of the matching list so that recvs for specific tags take precedence.
Ensure we don't try to send tcp mcast messsages to procs that haven't reported back yet

This commit was SVN r23491.
2010-07-23 19:31:34 +00:00
Ralph Castain
95a908cb27 Provide run-time control over heartbeat
This commit was SVN r23490.
2010-07-23 17:11:12 +00:00
Ralph Castain
dfdb1eca0c Remove windows dist file requirement
This commit was SVN r23480.
2010-07-23 01:23:15 +00:00
Ralph Castain
67027cfce4 Add a new "generic" ess module that supports direct launched MPI jobs so long as certain minimum envars are provided, regardless of launch environment.
This commit was SVN r23478.
2010-07-22 21:51:43 +00:00
Ralph Castain
e7719f0aa4 Update platform files, adjust sensor heartbeat module selection rules
This commit was SVN r23477.
2010-07-22 21:50:46 +00:00
Jeff Squyres
64cb8f5d7f Another round of man page cleanups from Debian mantainer Manuel
Prinz.  Many thanks!

This commit was SVN r23445.
2010-07-20 14:07:18 +00:00
Ralph Castain
f3e13b9766 Improve the efficiency by making the check for uniqueness of incoming hnp contact info much faster by including the hnp_uri in the job_family tracker object. Replace the global buffer storage with a quick routine to build the buffer from the jobfams array
This commit was SVN r23443.
2010-07-20 08:30:47 +00:00
Ralph Castain
f85e69b64b Continue enabling connect_accept across large numbers of independent jobs by replacing the hash tables with pointer_arrays to store routes to remote hnps to gain flexibility we'll need in the future.
This commit was SVN r23439.
2010-07-20 04:47:31 +00:00
Ralph Castain
248320b91a Enable connect_accept between multiple singleton jobs without the presence of an external rendezvous agent (e.g., ompi-server). This also enables connect_accept between processes in more than two jobs regardless of how they were started.
Create an ability to store the contact info for multiple HNPs being used to route between different job families. Modify the dpm orte module to pass the resulting store during the connect_accept procedure so that all jobs involved in the resulting communicator know how to route OOB messages between them.

Add a test provided by Philippe that tests this ability.

This commit was SVN r23438.
2010-07-20 04:22:45 +00:00
Ralph Castain
72525a5850 Allow the passing of NULL to orte_plm.kill_local_procs to match what we allow for the equivalent orte_odls call
This commit was SVN r23436.
2010-07-20 04:05:41 +00:00
Ralph Castain
ad5eaee4c6 Protect against NULL and provide additional resource check/error report
This commit was SVN r23432.
2010-07-19 18:33:32 +00:00
Ralph Castain
0355da6335 Cleanup pointer array addressing and protect against NULL
This commit was SVN r23431.
2010-07-19 18:30:04 +00:00
Ralph Castain
12cd07c9a9 Start reducing our dependency on the event library by removing at least one instance where we use it to redirect the program counter. Rolf reported occasional hangs of mpirun in very specific circumstances after all daemons were done. A review of MTT results indicates this may have been happening more generally in a small fraction of cases.
The problem was tracked to use of the grpcomm.onesided_barrier to control daemon/mpirun termination. This relied on messaging -and- required that the program counter jump from the errmgr back to grpcomm. On rare occasions, this jump did not occur, causing mpirun to hang.

This patch looks more invasive than it is - most of the affected files simply had one or two lines removed. The essence of the change is:

* pulled the job_complete and quit routines out of orterun and orted_main and put them in a common place

* modified the errmgr to directly call the new routines when termination is detected

* removed the grpcomm.onesided_barrier and its associated RML tag

* add a new "num_routes" API to the routed framework that reports back the number of dependent routes. When route_lost is called, the daemon's list of "children" is checked and adjusted if that route went to a "leaf" in the routing tree

* use connection termination between daemons to track rollup of the daemon tree. Daemons and HNP now terminate once num_routes returns zero

Also picked up in this commit is the addition of a new bool flag to the app_context struct, and increasing the job_control field from 8 to 16 bits. Both trivial.

This commit was SVN r23429.
2010-07-17 21:03:27 +00:00
Ralph Castain
a99e8cf132 Re-issue the read event so that multiple debugger attachments can occur
This commit was SVN r23421.
2010-07-15 15:35:26 +00:00
Jeff Squyres
c98729694a Fix a compile error in the xgrid plm. This does nothing to make it
work; it just now compiles on os x. 

This commit was SVN r23387.
2010-07-13 11:56:20 +00:00
Ralph Castain
570d19106b Allow singletons to use ompi-server for rendezvous via pubsub as well as comm_spawn without starting their own local daemons
This commit was SVN r23384.
2010-07-13 06:33:07 +00:00
Ralph Castain
8bb0c16c2f Add new tag
This commit was SVN r23382.
2010-07-13 06:29:13 +00:00
Ralph Castain
0b4081b162 The --output-filename option currently mixes the output from all processes with the same rank, but different jobids. Thus, the output from comm_spawn'd processes gets intermingled with their parents and each other.
Adding the jobid to the output filename solves the problem.

Thanks to Jody for pointing this out!

This commit was SVN r23380.
2010-07-12 21:43:49 +00:00
Ralph Castain
4c5ea3d1ef Allow the ALPS allocator to run if the BASIL_RESERVATION_ID has been set. Update show_help messages and comments to reflect this new scenario.
Thanks to Jerome Soumagne for the patch!

This commit was SVN r23374.
2010-07-12 15:42:25 +00:00
Ralph Castain
2babebf9c3 Revert some of a prior commit - there is no need for another flag to indicate one-sided termination of the orteds.
This commit was SVN r23372.
2010-07-10 03:33:01 +00:00
Ralph Castain
da61b69b15 Ensure we don't incorrectly return a non-zero exit code when normally terminating a slurm job.
Slurm, of course, must always be different...

This commit was SVN r23371.
2010-07-09 19:14:10 +00:00
Ralph Castain
5d2233c950 Add some new tags
This commit was SVN r23369.
2010-07-09 17:51:28 +00:00
Ralph Castain
2193ace464 Ensure the debugger symbols are loaded into orterun prior to orte_init so they are found by attaching debuggers.
This commit was SVN r23368.
2010-07-09 17:51:00 +00:00
Rolf vandeVaart
cc9b768fdb Fix typo and missing header - needed by Solaris
This commit was SVN r23367.
2010-07-09 14:33:13 +00:00
Shiqing Fan
c51c262e67 Relevant Windows fixes for r23360.
This commit was SVN r23363.

The following SVN revision numbers were found above:
  r23360 --> open-mpi/ompi@31295e8dc2
2010-07-07 16:58:16 +00:00
Ralph Castain
62a8b73f1a Correctly handle output sent to the group input channel
This commit was SVN r23362.
2010-07-07 14:17:48 +00:00
Ralph Castain
31295e8dc2 As discussed on today's telecon, reorganize the debugger attachment code in orte to better support efforts within the tool community aimed at exploring alternative methods. Move the debugger attachment code from the orterun directory to a new debugger framework. Organize the existing standard support code into an "mpir" component. Organize the current extensions for co-spawning debugger daemons into a separate "mpirx" component.
Since the MPIR symbols are now included in the ORTE library, remove duplicate declarations in OMPI and replace them with extern references to their ORTE instantiations.

This commit was SVN r23360.
2010-07-06 23:35:42 +00:00
Ralph Castain
98e7bc94f5 This belongs more properly in the ORCM repo now that we have MCA capability over there
This commit was SVN r23346.
2010-07-05 18:25:03 +00:00
Ralph Castain
dac1b9976e Make the default group channels bidirectional
This commit was SVN r23345.
2010-07-05 14:45:57 +00:00
Jeff Squyres
5bebdb97fa Need these header files for NetBSD. Thanks for the heads-up from
Aleksej Saushev.

This commit was SVN r23343.
2010-07-02 17:38:57 +00:00
Ralph Castain
ee4564c13b Add some useful debug
This commit was SVN r23339.
2010-07-02 03:35:47 +00:00
Ralph Castain
f3d90dfb8d Fully restore fault recovery, both at the individual process and daemon level.
NOTE: MPI fault recovery remains unavailable pending merge from Josh. This only covers ORTE-level processes.

This commit was SVN r23335.
2010-07-01 19:45:43 +00:00
Ralph Castain
7190415977 Fix JEFF's mistake - we cannot use orte_show_help if execv fails because we already closed all the file descriptors!
This commit was SVN r23334.
2010-07-01 19:41:26 +00:00
Ralph Castain
510ade9503 Do not use nodes that are flagged as down or do-not-use for this map. Modify error output to reflect possible reasons no nodes would be available
This commit was SVN r23333.
2010-07-01 19:39:31 +00:00
Ralph Castain
81a65f2c67 Define a new node state
This commit was SVN r23332.
2010-07-01 19:38:23 +00:00
Ralph Castain
f5548b8e0f remove a potential locking conflict, and let emacs go ahead and reformat the function (sigh)
This commit was SVN r23331.
2010-07-01 19:37:53 +00:00
Ralph Castain
d463aec2f6 Don't try to send to dead daemons, keep accounting straight so we don't hang
This commit was SVN r23330.
2010-07-01 19:37:02 +00:00
Ralph Castain
dd85689560 Cleanup pointer array addressing
This commit was SVN r23329.
2010-07-01 19:33:10 +00:00
Ralph Castain
26fbae447e Don't try to forward input when we already ordered shutdown. Check return codes on sends
This commit was SVN r23328.
2010-07-01 19:32:08 +00:00
Ralph Castain
3237b9ec87 Print a nice error message when a daemon fails, and exit with a non-zero status
This commit was SVN r23314.
2010-06-28 16:38:54 +00:00
Ralph Castain
a1ea6bc130 Ignore debugger daemon termination status - we don't care how they died.
This commit was SVN r23306.
2010-06-26 03:08:50 +00:00
Ralph Castain
099c3aad97 Fix a major foopah that broke debugger attach. With the revisions in updating proc state, we dropped the recording of each proc's pid. Thus, attaching debuggers would find a proctable whose pids all equal 0.
This required modification of the errmgr.update_state API so the pid could be passed in to the function that could update the proper data record(s). All calls to that API have been updated as well, but I obviously couldn't test them all.

Thanks to Dong Ahn (LLNL) for catching this problem!

Also fixed debugger daemon cospawn, both for initial launch and attach-while-running modes. Tested and verified on rsh and slurm.

This commit was SVN r23300.
2010-06-24 05:13:53 +00:00
Shiqing Fan
681df0089b Add a few new files into the tarball.
This commit was SVN r23297.
2010-06-22 16:45:56 +00:00
Ralph Castain
8b2a682fba Return a silent error when -do-not-launch is given
This commit was SVN r23291.
2010-06-22 01:06:10 +00:00
Shiqing Fan
2e5e9f0a03 Fix a wrong windows path in hpn_contack, which causes problems when looking up in the session directories. Add two more ess module for Windows.
This commit was SVN r23286.
2010-06-21 09:47:33 +00:00
Ethan Mallove
fc37e408c2 Avoid SEGV in case rsh/ssh is not in PATH (refs trac:1490)
This commit was SVN r23278.

The following Trac tickets were found above:
  Ticket 1490 --> https://svn.open-mpi.org/trac/ompi/ticket/1490
2010-06-17 14:58:09 +00:00
Ralph Castain
1e90b91b84 Unset envars we set during initialization so we leave environ intact after orte_finalize.
Thanks to Damien Gunter for pointing it out.

This commit was SVN r23277.
2010-06-17 13:42:21 +00:00
Ralph Castain
628ffd1d6e Make the mcast channel assignments unsigned ints so they can be used as array indices. Assign input/output channels for apps. Cleanup some bugs in open_channel
This commit was SVN r23275.
2010-06-16 19:40:59 +00:00
Ralph Castain
6cbe947810 Modify the multicast scheme so that applications have separate input and output channels to avoid cross-talk. Update the multicast test to conform.
This commit was SVN r23271.
2010-06-15 03:50:31 +00:00
Ralph Castain
da43547983 Don't define the active_jobid until -after- the job has been setup.
Cleanup references to pointer_array objects

This commit was SVN r23250.
2010-06-09 02:16:05 +00:00
Jeff Squyres
f1a7b5cc33 Make "processor affinity not supported" error message a little better:
* Remove OPAL_ERR_PAFFINITY_NOT_SUPPORTED; fit it into the generic
   OPAL_ERR_NOT_SUPPORTED case.
 * When odls_default detects that processor affinity is not supported,
   it prints a specific message about it, and then it suppressed a
   generic HNP help message that would normally follow it (i.e., it's
   easier to have the "processor affinity is not supported" show_help
   message last).
 * Use some symbolic names in odls_default instead of fixed int's,
   just for slight readability improvements in the code.
 * Introduce orte_show_help_suppress(), which gives the ability to
   suppress any future showings of any arbitrary show_help() message.
   This is useful if you display message X and want to suppress
   message Y.  This suppression *only* works in environments where
   orte_show_help() does coalescing.

This commit was SVN r23249.
2010-06-08 20:16:07 +00:00
Ralph Castain
e52a54183f Let max restarts be associated with an app_context instead of a job so that individual apps can have different values. Default to a single job-level value
This commit was SVN r23248.
2010-06-07 14:21:08 +00:00
Ralph Castain
799a77a187 Some updates to the routed-cm module so it properly supports the tcp rmcast module
This commit was SVN r23247.
2010-06-07 14:19:32 +00:00
Ralph Castain
bd045468e5 Let apps use the ess cm module too...
This commit was SVN r23246.
2010-06-07 14:16:34 +00:00
Ralph Castain
ec7b5dae2b Add missing include file
This commit was SVN r23245.
2010-06-07 14:15:25 +00:00
George Bosilca
f453265de2 Only call gettimeofday once.
This commit was SVN r23235.
2010-06-02 09:44:37 +00:00
Ralph Castain
69410f2a87 Ensure that we report the state on debugger daemon co-launch so that the spawn properly releases
This commit was SVN r23233.
2010-06-01 23:23:00 +00:00
Ralph Castain
b60c369489 Add missing rml tag
This commit was SVN r23232.
2010-06-01 22:58:23 +00:00
Shiqing Fan
2697a37363 Use the correct type for IO vector base.
This commit was SVN r23229.
2010-06-01 15:40:11 +00:00
Ralph Castain
36e6c11c5e Little cleanup
This commit was SVN r23211.
2010-05-27 02:49:09 +00:00
Ralph Castain
4ce07ace61 Allow the user to set the send/recv buf size for udp. Don't declare existing nb recvs to be an error.
This commit was SVN r23210.
2010-05-26 14:29:36 +00:00
Ralph Castain
ab6e06f5b3 Reorganize the rmcast code to capture common code elements. Increase max msg size for spread and udp transports. Cleanup the spread configuration doc.
This commit was SVN r23207.
2010-05-25 22:36:57 +00:00
Ralph Castain
02cc0cde83 Only activate this module if specifically requested
This commit was SVN r23203.
2010-05-24 18:42:32 +00:00
Abhishek Kulkarni
f04dcffecd Wrap the connection failed check with a SOS macro to extract the native error code.
This commit was SVN r23202.
2010-05-23 16:42:08 +00:00
Ralph Castain
73ebb748bb Ignore comm failures when shutting down orteds
This commit was SVN r23201.
2010-05-23 02:57:03 +00:00
Ralph Castain
e8f98661bb Fix a couple of plm modules that were calling a stale function
This commit was SVN r23200.
2010-05-23 02:55:47 +00:00
Ralph Castain
7c43d6c0f5 Don't drop a core file when we abort due to a lost connection
This commit was SVN r23199.
2010-05-22 18:09:40 +00:00
Jeff Squyres
fec7918eea Some paffinity functions had their return status overloaded:
* If < 0, it's an OPAL_ERR_* value
 * If >= 0, it's the actual output value of the function

This is problematic for the OPAL_SOS stuff.  This commit changes those
functions to always return OPAL_* statuses and send the output value
back through output parameters (like 95% of the rest of the code
base).  This avoids the confusion with OPAL_SOS stuff and makes
paffinity work again (e.g., mpirun --bind-to-core ...).

I updated all paffinitiy modules for the new function signatures, and
bumped the paffinity API version up to 2.0.1.  I don't think the
version change will matter, though, because we'll be introducing
support for hardware threads soon, which will either bump the
paffinity version again or we'll replace paffinity with 
a new framework.

This commit was SVN r23197.
2010-05-21 16:55:28 +00:00
Ethan Mallove
57eee4d75c * Can't put var declarations in the middle of code
* Use OBJ_RELEASE on data that was OBJ_NEW'd
 * Limit single-line char width
 * Use ORTE_ERR_BAD_PARAM on a rankfile typo, not ORTE_ERR_SILENT
 * Add copyright

This commit was SVN r23196.
2010-05-21 15:30:38 +00:00
Shiqing Fan
857f1669e2 Solve a few compilation problems on Windows.
This commit was SVN r23193.
2010-05-21 14:30:15 +00:00
Ralph Castain
aaaeea6f17 Once again, fix the blasted rank_file mapper. I can't guarantee that I fixed it correctly, but at least now it compiles!
This commit was SVN r23190.
2010-05-21 09:46:42 +00:00
Ethan Mallove
e751f3c21c Add a check for a duplicate rank assignment in the rankfile parser (Fixes trac:2414)
This commit was SVN r23186.

The following Trac tickets were found above:
  Ticket 2414 --> https://svn.open-mpi.org/trac/ompi/ticket/2414
2010-05-20 18:38:03 +00:00
Ralph Castain
ef3c88cbd2 If we have ordered jobs to terminate, then we should ignore comm_failed reports from daemons as they may be dropping out
This commit was SVN r23185.
2010-05-20 12:37:09 +00:00
Ralph Castain
05e05089b8 Ignore failed comm connections if it is our connection that failed
This commit was SVN r23184.
2010-05-20 03:13:09 +00:00
Abhishek Kulkarni
abe13d802c Silence warnings by commenting out unused functions in the "hnp" notifier component.
This commit was SVN r23181.
2010-05-19 22:46:05 +00:00
Abhishek Kulkarni
118ce0e166 OMPI FTB component updates
* register FTB events from an event schema file
  * define more FTB events
  * minor fixes

This commit was SVN r23180.
2010-05-19 22:05:06 +00:00
Ralph Castain
c7d7a18318 Little more cleanup from SOS
This commit was SVN r23175.
2010-05-19 16:28:58 +00:00
Josh Hursey
f57e73d4e5 add a few more missing SOS includes
This commit was SVN r23168.
2010-05-18 15:00:07 +00:00
Rolf vandeVaart
cdd2d09c69 Fix broken compile.
This commit was SVN r23167.
2010-05-18 12:43:21 +00:00
Abhishek Kulkarni
afbe3e99c6 * Wrap all the direct error-code checks of the form (OMPI_ERR_* == ret) with
(OMPI_ERR_* = OPAL_SOS_GET_ERR_CODE(ret)), since the return value could be a
 SOS-encoded error. The OPAL_SOS_GET_ERR_CODE() takes in a SOS error and returns
 back the native error code.

* Since OPAL_SUCCESS is preserved by SOS, also change all calls of the form
  (OPAL_ERROR == ret) to (OPAL_SUCCESS != ret). We thus avoid having to
  decode 'ret' to get the native error code.

This commit was SVN r23162.
2010-05-17 23:08:56 +00:00
Abhishek Kulkarni
f7f4dd87ab Change ORTE_ERR_LOG macro to make it SOS-aware.
If the logged error is an SOS-encoded error, use the SOS log function otherwise
use the existing errmgr log function.

This commit was SVN r23161.
2010-05-17 23:02:13 +00:00
Abhishek Kulkarni
9c5860706f Merge improvements to the "notifier" framework from the OPAL SOS and the ORTE WDC mercurial branches into the SVN trunk.
A brief description of the improvements can be found at
https://svn.open-mpi.org/trac/ompi/wiki/ORTEWDC#ChangesdonetotheORTEnotifier

This commit was SVN r23157.
2010-05-17 22:48:05 +00:00
Abhishek Kulkarni
f5b9bc4ff1 Add a new "HNP" component to the notifier framework.
This component proxies notification messages up to the HNP.  This
component runs in both the HNP and non-HNP processes for ease of
selection (e.g., so you can "--mca notifier hnp" (vs. "--mca
notifier hnp,non_hnp").  It auto-detects where it is running and
does the Right Thing -- if it's in the HNP process, it sets up to
receive incoming proxied messages.  If it's not in the HNP, then it
proxies all messages to the HNP.

This commit was SVN r23156.
2010-05-17 22:43:43 +00:00
Abhishek Kulkarni
197ec7586d Add a new "file" component to the notifier framework.
When this component is selected, the notification messages are sent to a file. 
The file can be a plain file or stdout or stderr.

The MCA parameter "notifier_file_name" can be used to specify the suffix of the file the notification messages should be sent to.
The default suffix is "wdc" and the file full name is "output-wdc".

This commit was SVN r23155.
2010-05-17 22:39:52 +00:00
Ralph Castain
7e6985edbf Cleanup warnings
This commit was SVN r23150.
2010-05-16 20:23:26 +00:00
Ralph Castain
de85049477 Cleanup warnings
This commit was SVN r23149.
2010-05-16 20:22:18 +00:00
Ralph Castain
88f5217a12 Cleanup the debugger daemon co-launch code and add an ability to test it. Implement ability to co-launch debugger daemons upon attach to a running job for jobs launched under rsh, slurm, and tm environments (others can easily be added if desired).
Add new mca params to test:

orte_debugger_test_daemon: Name of the executable to be used to simulate a debugger colaunch
orte_debugger_test_attach: Test debugger colaunch after debugger attachment

To test co-launch at job start, just set the orte_debugger_test_daemon param.

To test co-launch upon attach:
set orte_debugger_test_daemon
set orte_debugger_test_attach=1
set orte_enable_debug_cospawn_while_running=1
set orte_debugger_check_rate=<N> - defines the number of seconds to wait before "checking" for a debugger attaching

Added a "debugger" program to orte/test/mpi that just spins to simulate a debugger daemon.

This commit was SVN r23144.
2010-05-14 18:44:49 +00:00
Ralph Castain
b9f0615727 Correct some logic for tracking launch progress
This commit was SVN r23122.
2010-05-12 18:39:10 +00:00
Ralph Castain
7ce34223f1 Per off-list discussion, implement the new OMPI exit status policy (soon to be on wiki) and further cleanup error reporting to cover new cases.
Implement process migration when failed nodes are detected. Some testing still required

This commit was SVN r23121.
2010-05-12 18:11:58 +00:00
Ralph Castain
306533fdb8 Replace a missing line that shutdown a peer that failed comm.
This commit was SVN r23120.
2010-05-12 18:09:35 +00:00
Ralph Castain
871f445848 Ignore nodes that are "down" when generating maps
This commit was SVN r23119.
2010-05-12 18:08:40 +00:00
Ralph Castain
4bd25f587c Begin handling the case of lost connections by having the OOB report it to the errmgr instead of the routed framework. Add an "app" component to t
he errmgr framework so that it can decide how to respond - which for now at least is just to check for lifeline and abort if so.

Add a new error constant to indicate that the error is "unrecoverable" so the oob can know it needs to abort.

This commit was SVN r23112.
2010-05-11 00:34:12 +00:00