1
1
Граф коммитов

1334 Коммитов

Автор SHA1 Сообщение Дата
Ralph Castain
9cebe0ca96 Ckpt the bproc support. All compiles now except for PLM module
This commit was SVN r18744.
2008-06-26 03:48:22 +00:00
Brian Barrett
cbd6749c22 Move the lock initialization back to orte_init so that the finalize lock
is properly initialized and available in all cases (like ompi_info, where
the ess is never actually initialized).  Fixes trac:1364.

This commit was SVN r18733.

The following Trac tickets were found above:
  Ticket 1364 --> https://svn.open-mpi.org/trac/ompi/ticket/1364
2008-06-25 03:18:37 +00:00
Ralph Castain
578d1c15c6 Allow the ESS to return the hostname and arch for a specified daemon instead of just for application procs. Uses the same API - just need to detect that the specified proc is a daemon and lookup its corresponding node in the nidmap.
This commit was SVN r18722.
2008-06-24 17:53:10 +00:00
Ralph Castain
17fcd72b5d Restore bproc code - if someone wants to maintain it, then more power to them...but it would definitely be easier if the old code is in the trunk. This is all .ompi_ignore'd except for me so I can play with making it compile again in my copious free time.
This commit was SVN r18716.
2008-06-24 01:27:22 +00:00
Ralph Castain
3e61a3f92e Sandbox for next-gen launch
This commit was SVN r18715.
2008-06-24 01:25:51 +00:00
Ralph Castain
f799ea225f Orterun creates a "clean" copy of its environment for use in launching procs. This includes properly setting LD_LIBRARY_PATH and PATH, among other things. Unfortunately, our PLM modules were using the local environ instead of the saved copy, thus missing a number of things that really should have been included. From what I see, we got away with the error because the PLM's were duplicating all that setup logic themselves - I'll clean this up over the next few days.
Meantime, correct the PLM's so they use the correct environ for launching.

This commit was SVN r18713.
2008-06-23 22:39:36 +00:00
Ralph Castain
0fa9d88009 Set $PWD for the application proc to match the cwd. If the user specifies a working dir via -wdir, this ensures that the enviro variable matches what they get from getcwd. Note that any subsequent calls to chdir in the user's program will break that equivalence - we can only ensure it starts out matching!
This commit was SVN r18709.
2008-06-23 18:25:41 +00:00
Ralph Castain
acbcbb81b5 Add some debugging output to the modex set_proc_attr function to see what is being added to the modex
This commit was SVN r18708.
2008-06-23 18:24:08 +00:00
Ralph Castain
c693d3a5d1 I hadn't honestly considered before that an MPI process might attempt to call functions in the routed framework intended solely for daemons and HNPs. By design, MPI processes are not allowed to route RML/OOB messages, and hence the routed module in an MPI process has no knowledge whatsoever of how a message will reach its destination (except in the direct module). Thus, it has no way to return a valid routing tree, update a routing tree, or get wireup info.
This commit ensures that attempts to access information that is unknowable or undefined returns appropriate invalid or not_supported values to avoid unexpected behavior and/or segfaults.

This commit was SVN r18692.
2008-06-20 03:26:13 +00:00
Ralph Castain
174b9f1482 Ensure this module works in heterogeneous environments.
Note: this module is under development, which is why it is not set as the default. Use at your own risk!

This commit was SVN r18688.
2008-06-19 19:40:47 +00:00
Josh Hursey
b78ae13bf3 add back a missing a header taken away in r18664
This commit was SVN r18682.

The following SVN revision numbers were found above:
  r18664 --> open-mpi/ompi@0532d799d6
2008-06-19 16:08:27 +00:00
Ralph Castain
3b5e80fa61 Shift responsibility for preconnecting the oob to the orte routed framework, which is the only place that knows what needs to be done. Only the direct module will actually do anything - it uses the same algo as the original preconnect function.
This commit was SVN r18677.
2008-06-19 13:48:26 +00:00
Ralph Castain
b56f8ced4f Ensure params are registered prior to parsing global cmd line options in orterun so that debugger options are properly captured and acted upon.
Ensure that routes to remote procs are set on the HNP before completing launch so that the debugger message can be sent. Solves a race condition that can exist in those environments where the HNP does not have local procs.

This commit was SVN r18674.
2008-06-19 02:58:14 +00:00
Ralph Castain
955d117f5e Add a new grpcomm module that mimics the old 1.2 behavior - it -always- does a modex because it always includes the architecture. Hence, we called it "blind-and-dumb" since it doesn't look to see if this is required - moniker of "bad". :-)
Update the ESS API so we can update the stored arch's should the modex include that info. Update ompi/proc to check/set the arch for remote procs, and add that function call to mpi_init right after the modex is done.

Setup to allow other grpcomm modules to decide whether or not to add the arch to the modex, and to detect if other entries have been made. If not, then the modex can just fall through. Begin setting up some logic in the "basic" module to handle different arch situations.

For now, default to the "bad" module so we will work in all situations, even though we may be sending around more info than we really require.

This fixes ticket #1340

This commit was SVN r18673.
2008-06-18 22:17:53 +00:00
Ralph Castain
282a220e7e Update the debugger interface per email thread with Jeff and Brian. Handoff to them for final test and validation
This commit was SVN r18670.
2008-06-18 15:28:46 +00:00
George Bosilca
8e7c35e76c These symbols are only available via the module/component structure, so they
don't have to be globally visible.

This commit was SVN r18666.
2008-06-18 08:20:02 +00:00
Ralph Castain
0532d799d6 Complete implementation of the --without-rte-support configure option. Working with Brian, this has been tested on RedStorm.
Some minor changes to help facilitate debugger support so that both mpirun and yod can operate with it. Still to be completed.

This commit was SVN r18664.
2008-06-18 03:15:56 +00:00
Ralph Castain
a87aa442e3 Remove last remaining reference to iof_flush - it was #if'd out anyway. The existing flush code appears to have several critical problems. Given the impending rework of the IOF subsystem, there is no point in trying to fix it here.
This commit was SVN r18649.
2008-06-11 16:25:46 +00:00
Ralph Castain
f9d809748c Glad someone found that last error - caused me to review the code and find a couple of other cleanups! Nothing major, but just ensure that things flow smoothly since we had a "shadowed" variable.
This commit was SVN r18643.
2008-06-10 19:15:59 +00:00
Camille Coti
67cd1849f7 *map was still NULL in the else statement, inducing a segmentation fault when a field of the structure was accessed to.
This commit was SVN r18642.
2008-06-10 19:00:57 +00:00
Ralph Castain
1a422995ae Fix two Coverity complaints CID 813 (value defined and not used) and 1039 (resource leak). While doing so, found and fixed another less obvious memory leak.
This commit was SVN r18641.
2008-06-10 17:53:28 +00:00
Brian Barrett
4127bd0dcc fix two other mistakes in the cnos ess
This commit was SVN r18632.
2008-06-09 22:28:26 +00:00
George Bosilca
f72ab90b16 Allow xgrid to compile again.
This commit was SVN r18631.
2008-06-09 21:51:41 +00:00
Brian Barrett
11cd3a7cba Fix problem where local rank always had different architecture than remote
ranks on Red Storm

This commit was SVN r18630.
2008-06-09 21:46:03 +00:00
Ralph Castain
c13cadc3c7 Refs trac:1255
This commit repairs the debugger initialization procedure. I am not closing the ticket, however, pending Jeff's review of how it interfaces to the ompi_debugger code he implemented. There were duplicate symbols being created in that code, but not used anywhere. I replaced them with the ORTE-created symbols instead. However, since they aren't used anywhere, I have no way of checking to ensure I didn't break something.

So the ticket can be checked by Jeff when he returns from vacation... :-)

This commit was SVN r18625.

The following Trac tickets were found above:
  Ticket 1255 --> https://svn.open-mpi.org/trac/ompi/ticket/1255
2008-06-09 20:34:14 +00:00
Ralph Castain
bf5c34d10a The rsh launcher is one place where multi-word MCA params would have to be passed via the orted cmd line. In such a case, we have to explicitly include quote marks about the param value. Add that capability here.
This commit fixes trac:1200

This commit was SVN r18621.

The following Trac tickets were found above:
  Ticket 1200 --> https://svn.open-mpi.org/trac/ompi/ticket/1200
2008-06-09 19:07:19 +00:00
Ralph Castain
9613b3176c Effectively revert the orte_output system and return to direct use of opal_output at all levels. Retain the orte_show_help subsystem to allow aggregation of show_help messages at the HNP.
After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach.

I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive.

This commit was SVN r18619.
2008-06-09 14:53:58 +00:00
Pak Lui
caac0e0182 Add in a couple missing ones from r18611 for all tm users out there...
This commit was SVN r18615.

The following SVN revision numbers were found above:
  r18611 --> open-mpi/ompi@7bee71aa59
2008-06-06 22:53:43 +00:00
Ralph Castain
b65eb54ea2 Cut out a new iof pull - that capability isn't ready yet for the trunk, but will be coming shortly
Thanks to Pak for letting me know...

This commit was SVN r18614.
2008-06-06 21:24:15 +00:00
Pak Lui
7f7777a538 Check for NULL in prefix_dir.
This commit fixes trac:1337.

This commit was SVN r18612.

The following Trac tickets were found above:
  Ticket 1337 --> https://svn.open-mpi.org/trac/ompi/ticket/1337
2008-06-06 19:55:01 +00:00
Ralph Castain
7bee71aa59 Fix a potential, albeit perhaps esoteric, race condition that can occur for fast HNP's, slow orteds, and fast apps. Under those conditions, it is possible for the orted to be caught in its original send of contact info back to the HNP, and thus for the progress stack never to recover back to a high level. In those circumstances, the orted can "hang" when trying to exit.
Add a new function to opal_progress that tells us our recursion depth to support that solution.

Yes, I know this sounds picky, but good ol' Jeff managed to make it happen by driving his cluster near to death...

Also ensure that we declare "failed" for the daemon job when daemons fail instead of the application job. This is important so that orte knows that it cannot use xcast to tell daemons to "exit", nor should it expect all daemons to respond. Otherwise, it is possible to hang.

After lots of testing, decide to default (again) to slurm detecting failed orteds. This proved necessary to avoid rather annoying hangs that were difficult to recover from. There are conditions where slurm will fail to launch all daemons (slurm folks are working on it), and yet again, good ol' Jeff managed to find both of them.

Thanks you Jeff! :-/

This commit was SVN r18611.
2008-06-06 19:36:27 +00:00
Josh Hursey
1de50b523c Fix some Coverity 'Event set_but_not_used' highlights.
Thanks to Jeff for bringing them to my attention.

This commit was SVN r18606.
2008-06-06 14:38:41 +00:00
Jeff Squyres
d3795d7a34 Fix CID 987: remove unused variable.
This commit was SVN r18598.
2008-06-05 20:17:02 +00:00
Ralph Castain
332e6c89ab Modify the slurm launcher so that the kill-on-bad-exit behavior is not "on" by default. Instead, only turn it "on" if the plm_slurm_detect_failure mca param is set to something non-zero
This commit was SVN r18588.
2008-06-04 23:59:53 +00:00
Ralph Castain
0da811ce79 Initial work on xml support - allocation and job map outputs completed. More to come.
This commit was SVN r18587.
2008-06-04 20:53:12 +00:00
George Bosilca
25ae9c12e6 Silence few warnings.
This commit was SVN r18568.
2008-06-03 19:58:40 +00:00
George Bosilca
fa89d299bf Silence the Obj-C compiler.
This commit was SVN r18567.
2008-06-03 19:24:17 +00:00
Ralph Castain
c992e99035 Remove the tags from orte_output_open and the filtering operation from orte_output - this will be handled differently to improve the XML output interface
This commit was SVN r18557.
2008-06-03 14:24:01 +00:00
Ralph Castain
95578b0528 Fix single-node operations so that the HNP correctly exits when the job completes
This commit was SVN r18556.
2008-06-03 14:23:04 +00:00
Ralph Castain
b456fb2d42 Upgrade the node/orted failure detection code to cover all environments. Use the native environment's capabilities where possible - e.g., SLURM detects orted failure and can report it. Elsewhere, use a heartbeat system to detect orted failure - e.g., for TM and rsh. Heart rate is set via mca param. The HNP checks for callback every 2*heartrate, declares orted failure if not seen in last 2*heartrate time.
Also detect orted failed-to-start by setting timeout on launch. Currently only used in TM launcher.

Neither detection is enabled by default, but are only active if heartrate is set and/or launch timeout is set. Exception for SLURM as orted failure is always detected and reported.

More info to come on devel list.

This commit was SVN r18555.
2008-06-02 21:46:34 +00:00
Shiqing Fan
af656b2b3d Fix some typing mistakes, make the sources compile again for Windows Visual Studio.
This commit was SVN r18542.
2008-05-29 15:27:43 +00:00
Ralph Castain
2b28bef15a Provide a "nicer" indication that we don't know the pid of the failed orted
This commit was SVN r18538.
2008-05-29 14:10:58 +00:00
Ralph Castain
72530f8fed Cleanly handle the failed start of an orted, or its unexpected failure after start. This commit will allow mpirun to exit cleanly when this occurs, and does a best-effort attempt to cleanup the mess. However, it still has two unresolved issues that need to be eventually addressed:
1. it depends upon the ability of the native environment to alert us that the orted has died/failed to start. I have included that support for SLURM, but other environments need to be done.

2. for some yet-to-be-determined reason, the message that tells the remaining daemons to "die" isn't getting out of the RML, even though no obvious blockage is standing in the way. Work will continue on resolving that problem. For now, the orteds appear to be exiting on their own quite nicely when they see their HNP "lifeline" disappear.

This represents the best-available fix for ticket #221 so I am closing that ticket at this time.

This commit was SVN r18536.
2008-05-29 13:38:27 +00:00
Ralph Castain
52fb773c6c Tell slurm to kill the job if an orted abnormally exits
This commit was SVN r18535.
2008-05-29 12:26:58 +00:00
Ralph Castain
e5e542ddcf Clarify an error message
This commit was SVN r18533.
2008-05-29 12:20:24 +00:00
Josh Hursey
4ac7016200 Make sure to check "opal_list_get_last" instead of "opal_list_get_end".
The former will return a valid item in the list, the latter will return an invalid item that marks the end of the list.

It was happending that when oversubscribing by way of an appfile we would cause a segv because we tried to interpret the invalid item returned by "opal_list_get_end" instead of a valid item. We would then try to write to unallocated memory.

This commit fixes trac:1279

This commit was SVN r18529.

The following Trac tickets were found above:
  Ticket 1279 --> https://svn.open-mpi.org/trac/ompi/ticket/1279
2008-05-28 19:37:20 +00:00
Ralph Castain
f76240e7cc Modify the nidmap utility to pass daemon vpids for nodes. In some mapping algo's, it is possible for nodes to be skipped. This results in daemon vpids that differ from the index of their respective node in the node array, causing the daemon to not recognize procs that it is supposed to launch.
This commit was SVN r18528.
2008-05-28 18:38:47 +00:00
George Bosilca
1eb1742225 Remove this left over dependency.
This commit was SVN r18508.
2008-05-27 16:57:40 +00:00
Ralph Castain
93d932aa0c Ensure that the display-map and display-allocation outputs get processed through the new OPAL filter framework by passing them through orte_output instead of using the opal_dss.dump function.
This commit was SVN r18507.
2008-05-27 15:46:21 +00:00
Ralph Castain
0b2b655de5 Initialize a variable so it can correctly be dealt with at shutdown - fixes trac:1312
This commit was SVN r18505.

The following Trac tickets were found above:
  Ticket 1312 --> https://svn.open-mpi.org/trac/ompi/ticket/1312
2008-05-27 14:53:24 +00:00