1
1
Граф коммитов

1696 Коммитов

Автор SHA1 Сообщение Дата
Ralph Castain
9d381a4ebf Add a '!' option to the xterm iof option to invoke the -hold feature of xterm.
Correct the orte-show-help file when a rank is out of bounds, and do that test where a wildcard doesn't get incorrectly flagged as out-of-bounds.

This commit was SVN r20398.
2009-02-02 15:06:23 +00:00
Ralph Castain
d207c17adf Fix a segv when an application isn't found - ensure we properly terminate.
This commit was SVN r20395.
2009-02-02 13:44:08 +00:00
Ralph Castain
c3261e1a05 Fix optimized builds
This commit was SVN r20394.
2009-02-01 20:58:17 +00:00
Ralph Castain
debf128e53 Ensure the static port array is correctly checked for size
This commit was SVN r20393.
2009-01-31 03:46:42 +00:00
Ralph Castain
2966206f58 Fix a race condition in the IOF and add some new user-requested features:
1. fix a race condition whereby a proc's output could trigger an event prior to the other outputs being setup, thus c ausing the IOF to declare the proc "terminated" too early. This was really rare, but could happen.

2. add a new "timestamp-output" option that timestamp's each line of output

3. add a new "output-filename" option that redirects each proc's output to a separate rank-named file.

4. add a new "xterm" option that redirects the output of the specified ranks to a separate xterm window.

This commit was SVN r20392.
2009-01-30 22:47:30 +00:00
Rolf vandeVaart
0704b98668 Add the ability to forward SIGTSTP (converted to SIGSTOP) and
SIGCONT to the a.outs.  By default, they are not forwarded and
the behavior remains as it has always been.  However, if one
runs with --mca orte_forward_job_control 1, then mpirun will
catch those two signals and forward them to the orteds which
will deliver them to the a.outs.  We have had requests for
this feature.

This commit was SVN r20391.
2009-01-30 18:50:10 +00:00
Ralph Castain
5e6d3ba289 Initial implementation of static ports. Provide an mca param to specify static port ranges to the OOB - can provide an
y combination of comma-separated values and ranges. Daemons will use the first port in the range, MPI procs will use the other ports in the range assuming that they know their node rank in time and enough ports were specified.

NOTE: this capability only works under specific conditions. I will outline more about this in a note to devel as the remainder of the implementation progresses. For now, the only environment where this works is slurm. The linear routed module has also been adjusted to work with static ports so that all messaging flows strictly through the topology, including the initial daemon callback - thus limiting the number of sockets opened by mpirun.

This commit was SVN r20390.
2009-01-30 18:31:43 +00:00
Jeff Squyres
35c5e28a8e Up to SVN r20383
This commit was SVN r20384.

The following SVN revision numbers were found above:
  r20383 --> open-mpi/ompi@e0638c84c8
2009-01-29 17:59:04 +00:00
Jeff Squyres
bb3d258562 Round up a few places where PATH_MAX was used instead of
OMPI_PATH_MAX.  Thanks to Andrea Iob for the bug report.

This commit was SVN r20360.
2009-01-27 22:57:50 +00:00
Ralph Castain
c92f906d7c Move the daemon collectives out of the ODLS and into the GRPCOMM framework. This removes the inherent assumption that the OOB topology is a tree, thus allowing different grpcomm/routed combinations to implement collectives appropriate to their topology.
This commit was SVN r20357.
2009-01-27 19:13:56 +00:00
Ralph Castain
0435108834 Improve the efficiency of the launch system by changing the outer loop to being over app_context, and adding a flag to the app_context so the daemon can record that "this app is on my node" when decoding the launch msg.
If the --wdir option is given, check to see if the user provided a relative path. If so, convert it to an absolute path. This is needed to maintain consistent behavior across environements. Some environments automatically chdir to your current working directory when launching the remote orted, while others (e.g., ssh) don't. This levels the playing field and reduces user surprise.

This commit was SVN r20342.
2009-01-25 12:39:24 +00:00
Ralph Castain
40b6ed4a40 Take another crack at fixing the -wdir problem. Move the context checking code down into just prior to launching each child app. This is necessary so that individual app context wdir options are respected. Also, ensure that we return to our "base" directory after each app is launched so that the relative positions of the wdir options for each app_context are with respect to our base directory, instead of the last wdir option.
Hopefully, this will pass the "BigRed test". :-)

This commit was SVN r20341.
2009-01-24 20:59:27 +00:00
Tim Mattox
c2d105a4d9 Refs trac:1763: Fix -wdir option
Reverted r20306 since the fix caused 100% failues on our !BigRed system.

See the comments on ticket #1763 for the details.

This commit was SVN r20339.

The following SVN revision numbers were found above:
  r20306 --> open-mpi/ompi@8c87e48721

The following Trac tickets were found above:
  Ticket 1763 --> https://svn.open-mpi.org/trac/ompi/ticket/1763
2009-01-24 15:04:47 +00:00
Ralph Castain
c6c5bc17a0 Add a new hierarchical collective grpcomm component that performs modex and barrier across the procs instead of the daemons. Modeled on the tuned collectives. Collective code is in grpcomm base for eventual use by the daemon-based components as well.
This commit was SVN r20337.
2009-01-23 21:57:51 +00:00
Ralph Castain
7154cbf2e0 Cleanup a couple of mis-labeled diagnostic outputs
This commit was SVN r20332.
2009-01-23 20:46:54 +00:00
Josh Hursey
04c69b8a82 Fixes for --preload-files and --preload-binary.
* Improved the error propagation from a backend orted
* Fixed a hang in orterun due to failed files transferred
* Fix the movement of files with relative path names
* Improved error messages when a file cannot be moved
* Move file checks to FileM instead of embedding then in the ODLS

This commit Refs trac:1770

This commit was SVN r20331.

The following Trac tickets were found above:
  Ticket 1770 --> https://svn.open-mpi.org/trac/ompi/ticket/1770
2009-01-23 15:32:24 +00:00
Josh Hursey
d066c67b53 We need to update both context->app and context->argv[0] with the new path when we use --preload-binary. This keeps orte from checking the wrong path later in the odls [orte_util_check_context_app() called from odls_base_default_setup_fork()].
Refs trac:1770

This commit was SVN r20321.

The following Trac tickets were found above:
  Ticket 1770 --> https://svn.open-mpi.org/trac/ompi/ticket/1770
2009-01-22 19:18:36 +00:00
Ralph Castain
47740d1e87 Get the inequality the correct way!
This commit was SVN r20319.
2009-01-22 16:33:07 +00:00
Ralph Castain
f6ba4f6f30 Per discussion with Jeff, an invalid local rank value should never occur - if it does, it could be indicative of deeper problems in the launch procedure. Thus, rather than allowing the launch to proceed, let's abort.
This commit was SVN r20312.
2009-01-22 00:52:46 +00:00
Ralph Castain
8c87e48721 Fix a user-reported bug whereby the -wdir option would only be applied from the last app_context.
This commit was SVN r20306.
2009-01-21 15:52:12 +00:00
Ralph Castain
5d9de3326c Check for valid local/node ranks before using the returned values
This commit was SVN r20304.
2009-01-21 00:54:50 +00:00
Ralph Castain
a6a7335694 Catch a potential bug spanning several ESS modules. The node_rank and local_rank types were changed to uint16_t, however the modules returned UINT8_MAX as an "invalid" value. To clean this up, define an INVALID value for these types, and change the various modules so they return this value to indicate an invalid response.
This commit was SVN r20303.
2009-01-21 00:19:37 +00:00
Ralph Castain
4da9f53fa4 Implement the xml formatted output of stdout/err/diag. Force -tag-output if -xml is set.
This commit was SVN r20302.
2009-01-20 16:58:31 +00:00
Ralph Castain
88a0af9726 Revise the way we output resolved hostnames to make life easier for the Eclipse folks. Store aliases for individual nodes (only when requested to show resolved hostnames) and then report them out as part of the display-map option.
This commit was SVN r20284.
2009-01-15 18:11:50 +00:00
Ralph Castain
253a54df12 Shutdown the socket before closing for cleaner termination.
This commit was SVN r20283.
2009-01-15 18:06:01 +00:00
Ralph Castain
a9af219ba7 Fix CID 723: a pointless whine about not checking a return code
This commit was SVN r20274.
2009-01-14 19:06:36 +00:00
Josh Hursey
a9da2dada1 Remove some unused variables.
This commit was SVN r20270.
2009-01-14 17:28:40 +00:00
Tim Mattox
5b70160626 For two error conditions in the ras_loadleveler_module, output
the error code reported by loadleveler.  Also, clean up a
few more internal error messages.

This commit was SVN r20255.
2009-01-13 15:44:26 +00:00
Brian Barrett
d3310a5ad1 fixes to get compiling on Red Storm again
This commit was SVN r20252.
2009-01-12 22:30:00 +00:00
Ralph Castain
694008e9bb Fix a reported bug whereby keyboard entry to a remote proc was being lost after the first iteration. In other words, if an application has a proc reading stdin from the keyboard, and that proc is not co-located with mpirun, then the system would hang.
The problem was eventually traced to two bugs in the code:

1. the orted wasn't resetting the write event flag, thus preventing itself from turning it on again.

2. the HNP needed to check if the stdin was attached to tty or not before adding the delay for fairness. If it is attached to a tty, there is no need for the delay. This prevents some strangely slow typing response.

This patch needs to move to 1.3

This commit was SVN r20246.
2009-01-12 20:12:58 +00:00
Josh Hursey
1420c32a5d Update SnapC Local Coordinator in reaction to structure changes in r20228. The list of local children became more globalized so I needed to update the loop invariants appropriately.
This commit was SVN r20245.

The following SVN revision numbers were found above:
  r20228 --> open-mpi/ompi@007d68becc
2009-01-12 19:45:48 +00:00
Ralph Castain
2778c13fac Continue to refine the timing instrumentation to identify where launch time is being spent
This commit was SVN r20244.
2009-01-12 19:12:58 +00:00
Jeff Squyres
d1c6f3f89a * Fix a truckload of Cisco copyrights to be the same as the rest of
the code base.
 * Fix a few misspellings in other copyrights.

This commit was SVN r20241.
2009-01-11 02:30:00 +00:00
Tim Mattox
820b209564 Oops, forgot to update the copyright date range...
This commit was SVN r20239.
2009-01-09 19:04:52 +00:00
Tim Mattox
af45569366 Clean up some debugging output in the loadleveler ras module.
Error output strings were changed to be unique per code site.
They are still pretty meaningless to the user, but at least now
developers might be able to find which unique place in the code
reported which error.

This commit was SVN r20238.
2009-01-09 19:03:52 +00:00
Ralph Castain
c009b51ad3 Silence warning about signed vs unsigned comparisons
This commit was SVN r20237.
2009-01-09 16:01:03 +00:00
George Bosilca
78d856e04c Release resources when a job is completed. This allows us to correctly
count and load balance MPI-2 dynamic type of applications.

This commit was SVN r20236.
2009-01-08 21:21:54 +00:00
Ralph Castain
25f578a7d2 Continue to improve timing instrumentation. Add ability to store timing data directly to a file instead of just to stdout.
This commit was SVN r20229.
2009-01-08 14:27:52 +00:00
Ralph Castain
007d68becc Make the data on local children and their jobs available globally on both daemons and the HNP. This simply shifts the data structures from the ODLS base to the orte globals area to support subsequent movement of the daemon collective operations from the odls to the grpcomm framework. As that will be a larger change, it will be implemented on a branch and rolled over separately.
This commit was SVN r20228.
2009-01-08 14:25:56 +00:00
Ralph Castain
80fb98ae32 Cleanup the modex-less operations for efficiency. Have the component default to normal modex operations if modex-less isn't specified.
This commit was SVN r20220.
2009-01-07 15:00:26 +00:00
Ralph Castain
7818779760 Expose the nidmap and pidmap as orte globals so that components in other frameworks can access and/or manipulate them without forcing API modifications - modify the individual ess components that were affected so they use the global variables. Add a list of attributes to the nids for storing node-related data (e.g., modex attrs), and define a new object for that purpose.
Consolidate the nid/pid lookup code with the rest of the nid/pid code so that changes are easier to track. Add the ability to send cluster profile info as part of the nidmap. Cleanup the setup and teardown of the new global nidmap and pidmap objects.

This commit was SVN r20219.
2009-01-07 14:58:38 +00:00
Ralph Castain
9dbcee9110 Increase efficiency for modex-less launch by storing byte objects in the profile file
This commit was SVN r20206.
2009-01-05 21:46:12 +00:00
Ralph Castain
5f5d8ad231 CID 1139-1141: remove outdated variable from the various routed components
This commit was SVN r20201.
2009-01-05 15:09:54 +00:00
Ralph Castain
1bc125c0a7 CID 1131: cleanup a minor memory leak
This commit was SVN r20200.
2009-01-05 15:05:05 +00:00
Jeff Squyres
6d0d8848ac Fix CID 1129: Remove variable that is set but never used.
This commit was SVN r20194.
2009-01-03 15:39:51 +00:00
Jeff Squyres
e52ac6da40 Fix CID 1130: remove variable that is set but never used.
This commit was SVN r20193.
2009-01-03 15:37:00 +00:00
Ralph Castain
7787f84540 Per the earlier RFC and some discussion at the Dec ORTE design meeting, add the ompi-top tool and all its supporting infrastructure. This includes a new OPAL pstat framework and data type, currently with rather weak support for Mac OSX and pretty complete support for Linux. The Sun team promised to add Solaris support as well.
Also, per chat with Jeff, modified the Makefile.am's of a few orte tools so that they were consistent in the way we generate the ompi-equivalent cmds.

This commit was SVN r20165.
2008-12-22 20:23:05 +00:00
Ralph Castain
caa5771908 Don't force tools to dump core files when they abort
This commit was SVN r20159.
2008-12-20 23:24:36 +00:00
Ralph Castain
9f6c1b9d07 Per discussion at the Dec ORTE design meeting, add an "set_lifeline" API to the orte_routed framework. This allows the caller to define a "lifeline" process so that, if the connection to that lifeline is subsequently lost, the process will be terminated. This helps tools that connect to an mpirun to know when that mpirun completes and terminates.
This commit was SVN r20158.
2008-12-20 23:23:11 +00:00
Shiqing Fan
5ae5f0e173 - 4/4 commit for Windows Visual Studio and CCP support:
unnecessary clean up to non windows related files (within ifdef __WINDOWS__).

This commit was SVN r20111.
2008-12-10 21:13:27 +00:00
Shiqing Fan
8673f19f50 - 2/4 commit for Windows Visual Studio and CCP support:
changes to the already existing ccp components
  event/win32.c: merge old FD handling into new
  opal_installdirs_windows.c:fix the registry handling

This commit was SVN r20109.
2008-12-10 21:01:54 +00:00
Shiqing Fan
a5281f0434 - 1/4 commit for Windows Visual Studio and CCP support:
CMakeLists and .windows files.
  In contribs preconfigured and precompiled parts.

This commit was SVN r20108.
2008-12-10 20:59:20 +00:00
Ralph Castain
728a24c8ec After considerable patience and help with debugging/testing from Tim M and Jeff S, return a completed and pretty well tested patch of the IOF to the trunk. This commit includes the previously reverted r20074, r20068, and r20064, as well as changes to fix those commits.
Basically, the remaining problem turned out to be:

1. closing stdout/stderr during orte_finalize of mpirun

2. inadvertently setting up a write event on fd = -1

3. devising a scheme to more accurately track when the stdin write event was active vs closed so it only got released once

This passed prelim MTT testing by Jeff and Tim, but should soak for awhile before migrating to 1.3.

This commit was SVN r20106.

The following SVN revision numbers were found above:
  r20064 --> open-mpi/ompi@a07660aea8
  r20068 --> open-mpi/ompi@ec930d14a9
  r20074 --> open-mpi/ompi@2940309613
2008-12-10 20:40:47 +00:00
Ralph Castain
9d7cb82bba Modify the daemon cmd processor to relay and then process the cmd locally. We couldn't do this before due to the daemon's needing to update contact info prior to doing the relay. However, the new routed system plus the inclusion of the nidmap in the launch message now makes this possible.
It is a small launch performance improvement as now we relay the launch cmd across to the next daemon before taking the time to launch our own local procs. Still, it does allow more parallel operations during the launch procedure.

This commit was SVN r20104.
2008-12-10 19:18:36 +00:00
Josh Hursey
67ae66326c remove unused variable
This commit was SVN r20103.
2008-12-10 18:08:46 +00:00
Ralph Castain
1ace83c470 Enable modex-less launch. Consists of:
1. minor modification to include two new opal MCA params:
   (a) opal_profile: outputs what components were selected by each framework
       currently enabled for most, but not all, frameworks
   (b) opal_profile_file: name of file that contains profile info required
       for modex

2. introduction of two new tools:
   (a) ompi-probe: MPI process that simply calls MPI_Init/Finalize with
       opal_profile set. Also reports back the rml IP address for all
       interfaces on the node
   (b) ompi-profiler: uses ompi-probe to create the profile_file, also
       reports out a summary of what framework components are actually
       being used to help with configuration options

3. modification of the grpcomm basic component to utilize the
   profile file in place of the modex where possible

4. modification of orterun so it properly sees opal mca params and
   handles opal_profile correctly to ensure we don't get its profile

5. similar mod to orted as for orterun

6. addition of new test that calls orte_init followed by calls to
   grpcomm.barrier

This is all completely benign unless actively selected. At the moment, it only supports modex-less launch for openib-based systems. Minor mod to the TCP btl would be required to enable it as well, if people are interested. Similarly, anyone interested in enabling other BTL's for modex-less operation should let me know and I'll give you the magic details.

This seems to significantly improve scalability provided the file can be locally located on the nodes. I'm looking at an alternative means of disseminating the info (perhaps in launch message) as an option for removing that constraint.

This commit was SVN r20098.
2008-12-09 23:49:02 +00:00
Ralph Castain
e28210d0dc Revert r20074, r20068, and r20064: remove the IOF proc completion code pending further off-trunk work.
This commit was SVN r20089.

The following SVN revision numbers were found above:
  r20064 --> open-mpi/ompi@a07660aea8
  r20068 --> open-mpi/ompi@ec930d14a9
  r20074 --> open-mpi/ompi@2940309613
2008-12-09 17:11:59 +00:00
Ralph Castain
61c21d787d Add missing param in tm launcher
This commit was SVN r20087.
2008-12-09 13:31:33 +00:00
Ralph Castain
6e050bc78c Update the route when it comes from a different job family.
This fixes ticket #1699

This commit was SVN r20085.
2008-12-09 01:16:18 +00:00
Ralph Castain
ce4018efeb Take a step back on the slurm and tm launchers. Problems were occurring in the MTT runs, although not under non-MTT scenarios. Preserve the modified plm versions in new components that are ompi_ignored until we can resolve the problems.
This will allow for better MTT coverage until the problem can be better understood.

This commit was SVN r20083.
2008-12-09 00:32:04 +00:00
Ralph Castain
89792bbc72 May as well have the other "clean" outputs use the same channel
This commit was SVN r20082.
2008-12-08 19:37:22 +00:00
Ralph Castain
c2b18b363d Initialize a variable before use
This commit was SVN r20080.
2008-12-08 16:16:40 +00:00
Ralph Castain
2940309613 Attempt to solve a race condition showing up in some MTT runs. There were three entry points for proc termination info into the ODLS:
1. a direct callback from waitpid - this set the waitpid_fired flag

2. a notify event callback from the IOF - this set the iof complete flag

3. a message via the daemon cmd processor from the proc "de-registering" the sync, thus indicating it was going through MPI_Finalize.

The problem is that these could overlap, with the first two allowing the orted to declare the proc complete before the daemon had responded to #3.

This change forces all three events to flow through the daemon cmd processor, thus ensuring an ordered handling. I'm not certain this will solve the problem, but will await further MTT reports to see. Unfortunately, the problem doesn't show up on any manual or script-based tests I have been able to run, even when I duplicate the exact cmd that fails under MTT.

This commit was SVN r20074.
2008-12-05 04:20:00 +00:00
Ralph Castain
ec930d14a9 Ensure IOF tags are properly assigned to sinks and read events
This commit was SVN r20068.
2008-12-04 01:09:20 +00:00
Ralph Castain
a07660aea8 Bring over the IOF completion changes. This commit fixes the long-occurring problem whereby application procs could, under some circumstances, lose their final prints to stdout/err. The commit includes:
1. coordination of job completion notification to include a requirement for both waitpid detection AND notification that all iof pipes have been closed by the app

2. change of all IOF read and write events to be non-persistent so they can properly be shutdown and restarted only when required

3. addition of a delay (currently set to 10ms) before restarting the stdin read event. This was required to ensure that the stdout, stderr, and stddiag read events had an opportunity to be serviced in scenarios where large files are attached to stdin.

This commit was SVN r20064.
2008-12-03 17:45:42 +00:00
Josh Hursey
44109e0084 Fix the ft_event function in response to r20022. Also make the structure cleanup match the finalize() function a bit more closely.
This seems to fix the segv seen on process restart.

This commit was SVN r20051.

The following SVN revision numbers were found above:
  r20022 --> open-mpi/ompi@9a57db4a81
2008-12-02 21:18:32 +00:00
Ralph Castain
ff8e83ff3b Per request from IBM/Eclipse, provide MCA param to request output when nodes are resolved to a different nodename. This really only happens for the node that mpirun executes on, but they need the alert so they can do string matching of node names.
This commit was SVN r20032.
2008-11-24 19:57:08 +00:00
George Bosilca
7a30a98a89 Use the generic cast.
This commit was SVN r20028.
2008-11-24 15:52:36 +00:00
Ralph Castain
7213c109ac Revamp the TM plm module so that we detect orted termination without requiring a callback message by using the TM native capabilities. This allows TM to function with fully routed OOB comm, and to tell us what node failed to spawn a daemon.
This commit was SVN r20027.
2008-11-20 18:57:35 +00:00
Ralph Castain
9a57db4a81 To support comm_spawn in fully routed environments, daemons need to know the route to all procs in their job family. They already had this information, but were not retaining it. The infrastructure to do so has existed for some time - just never had the time to complete it.
This commit does that by ensuring that daemons retain knowledge of proc location for all procs in their job family. It required a minor change to the ESS API to allow the daemons to update their pidmaps as data was received. In addition, the routed modules have been updated to take advantage of the newly available info, and the encode/decode pidmap utilities have been updated to communicate the required info in the launch message.

This commit was SVN r20022.
2008-11-18 15:35:50 +00:00
Ralph Castain
89559396ea Resolve a race condition when running under a SLURM environment.
The slurm plm fork/exec's a call to srun to launch its daemons. When mpirun terminates, it then sends out a "terminate" command to those daemons. The daemons respond back to mpirun, and then exit.

If slurm itself is running on a slow network, and mpirun is running the OOB across a fast network, then it is possible for mpirun to receive notification of daemon termination and exit -before- the srun can complete its bookkeeping and declare the job as complete. When this happens, slurm becomes confused and loses state.

Mucho bad. :-/

This commit changes the termination logic so that mpirun will wait for srun to report complete before exiting. It also enables fully routed communications since it no longer requires daemons to report back that they are terminating, thus allowing the daemons to terminate asynchronously (thereby breaking routing paths).

This commit was SVN r20018.
2008-11-18 13:59:23 +00:00
Ralph Castain
68423f7544 Partially restore the iof changes - this repairs the initial observation of inconsistent and incomplete output
This commit was SVN r19999.
2008-11-14 20:36:18 +00:00
Ralph Castain
586334d1c8 Per discussion with Tim Mattox, reset the trunk to pre-19991 level for the iof only. I will shortly add a changeset that will repair the one known error where we were incorrectly closing the stdout/err/diag file descriptors when all we wanted to do was close stdin. I will leave out the changes associated with coordinating proc termination due to race conditions IU encounted during MTT testing. I have been unable to replicate those so far, but we hope to resolve it in the near future.
This commit was SVN r19998.
2008-11-14 20:22:36 +00:00
Ralph Castain
891630ae85 Handle a race condition between mpirun detecting stdin closed (and releasing the read event), and receiving an xon/xoff notice from a remote orted that detects proc termination and tells mpirun "don't send any more input - the proc is gone". This latter was necessary since we might have hung an infinite source of input on mpirun, while the proc terminated after some point in time.
This commit was SVN r19997.
2008-11-14 15:19:53 +00:00
Ralph Castain
101b6fdeb8 Cleanup a little on how we handle the stdin write when we encounter end-of-input. Ensure that mpirun handles it correctly if the proc receiving stdin is local to mpirun
This commit was SVN r19996.
2008-11-14 14:31:33 +00:00
Ralph Castain
875741a5e3 Don't set the stdin fd to -1 before calling the object destructor as that function calls event delete, which uses the fd as an index into the event array.
This commit was SVN r19994.
2008-11-13 19:34:29 +00:00
Ralph Castain
b8ae4604ed Correct the notifier default module to include the new added API
This commit was SVN r19993.
2008-11-13 18:03:41 +00:00
Ralph Castain
702fc7154c Remove stale function definition
This commit was SVN r19992.
2008-11-13 05:07:11 +00:00
Ralph Castain
555bbf0c02 Fix the iof race conditions wrt proc termination. This is comprised of two sections:
1. modify the iof to track when a proc actually closes all of its open iof output pipes. When this occurs, notify the odls that the proc's iof is complete. This is done via a zero-time event so that we can step out of the read event before processing the notification.

2. in the odls, modify the waitpid callback so it only flags that it was called. Add a function to receive the iof-complete notification, and a function that checks for both iof complete and waitpid callback before declaring a proc fully terminated. This ensures that we read and deliver -all- of the IO prior to declaring the job complete.

Also modified the odls call to orte_iof.close (and the component's implementation) so it only closes stdin, leaving the other io channels alone. This fixes the other half of the known problem.

This should fix the ticket on this subject, but I'll wait to close it pending further testing in the trunk.

This commit was SVN r19991.
2008-11-12 23:32:01 +00:00
Ralph Castain
26cd1c1955 Fix a typo and some formatting
This commit was SVN r19990.
2008-11-12 22:01:40 +00:00
Ralph Castain
ce26e3a2fb Update the notifier framework in prep for move to v1.3. Add an API to handle the case where error messages have been expressed via "show_help" so they can look similar to what was presented to users. Add three key calls in the openib btl to drop messages into syslog.
This will sit in trunk for a few days - would like to actually see some errors reported to syslog before moving the code to 1.3

This commit was SVN r19986.
2008-11-12 18:03:51 +00:00
Josh Hursey
d5c38c2601 fix some typos. should be moved to v1.3
This commit was SVN r19964.
2008-11-10 19:05:26 +00:00
Josh Hursey
077b3df7cc Fix C/R restart case by passing the correct address to the orte_ess_base_build_nidmap() function. This cropped up from r19866.
It does not look like this effects the v1.3 branch since r19866 has not moved to the release branch.

Thanks to Leonardo Fialho for reporting this and supplying a patch.

This commit was SVN r19961.

The following SVN revision numbers were found above:
  r19866 --> open-mpi/ompi@f54fda489e
2008-11-10 15:19:28 +00:00
Ralph Castain
5889dcd30b Fix a warning reported by Jeff that actually could cause singleton operations to fail. Ensure that the byte object used to init the job map for singleton's is properly initialized.
This commit was SVN r19957.
2008-11-08 01:09:06 +00:00
Jeff Squyres
f4ba25cf3c Remove linking components against ORTE and OPAL libs. This was
removed from all other components long ago; I'm not sure how these
survived.

This commit was SVN r19956.
2008-11-08 00:56:57 +00:00
Ralph Castain
25491628b8 Discovered while documenting the "preconnect" mca params that several of them didn't make sense any more. After chatting with Jeff, we agreed to the following:
1. register "mpi_preconnect_all" as a deprecated synonym for "mpi_preconnect_mpi"

2. remove "mpi_preconnect_oob" and "mpi_preconnect_oob_simultaneous" as these are no longer valid.

3. remove the routed framework's "warmup_routes" API. With the removal of the direct routed component, this function at best only wasted communications. The daemon routes are completely "warmed up" during launch, so having MPI procs order the sending of additional messages is simply wasteful.

4. remove the call to orte_routed.warmup_routes from MPI_Init. This was the only place it was used anyway.

The FAQs will be updated to reflect this changed situation, and a CMR filed to move this to the 1.3 branch.

This commit was SVN r19933.
2008-11-05 19:41:16 +00:00
Ralph Castain
b48bbec366 Cleanup modex logic to allow modex-less launch:
1. minor change in base_modex to only set modex_reqd when it -is- reqd

2. cleanup logic in grpcomm-basic module

This commit was SVN r19903.
2008-11-03 21:48:52 +00:00
Ralph Castain
6db5737779 Remove a couple of mutex vars that were defined and used - but never initialized. No clear way to initialize them, and that area of the code should never see threads anyway.
This commit was SVN r19889.
2008-11-03 17:23:10 +00:00
Kenneth Matney
c650ef58c5 Build requires prototypes, defined by "orte/util/nidmap.h".
This commit was SVN r19887.
2008-11-03 16:23:42 +00:00
Ralph Castain
55f52d7a4b Ensure we know how to route to a different job family when it connects to us
This commit was SVN r19885.
2008-11-03 14:25:14 +00:00
Ralph Castain
85bc7bb26a Minor cleanups:
* fix an if condition so that we do the right thing when procs local to mpirun output to stderr

* ensure that tools can handle relays of 0-byte output, indicating that a process closed that io channel

This commit was SVN r19884.
2008-11-03 14:03:08 +00:00
Ralph Castain
58fe779388 Remove double destruct to fix segv when ctrl-c is used to terminate job
This commit was SVN r19875.
2008-11-02 02:25:20 +00:00
George Bosilca
d23fe1bb10 Include Ralph's suggestions, i.e. keep the hnp and orted management in sync.
This commit was SVN r19872.
2008-11-01 00:39:46 +00:00
George Bosilca
9528d33e90 Nothing relevant, few indentations and replace tab by spaces.
This commit was SVN r19870.
2008-10-31 22:24:52 +00:00
George Bosilca
ebe87d1842 Apply some suggestions from Ralph and avoid a pretty nasty race condition on the close of the fd.
The problem was that we close the same fd twice, and that meantime the fd could have been reassigned
to some other file or socket.

This commit was SVN r19869.
2008-10-31 22:23:53 +00:00
George Bosilca
9f17d1d67d Allow xgrid to compile with the changes from 19866.
This commit was SVN r19868.
2008-10-31 21:56:53 +00:00
Ralph Castain
f54fda489e This is a first step towards supporting fully-routed OOB communications:
1. remove direct routed module (hooray!)

2. add radix tree routed module (binomial remains default)

3. remove duplicate data storage - orteds were storing nidmap and pidmap data in odls, everyone else in ess

4. add ess APIs to update nidmap, add new pidmap - used only by orteds for MPI-2 support

5. modify code to eliminate multiple calls to orte_routed.update_route that recreated info already in ess pidmap. Add ess API to lookup that info instead. Modify routed modules to utilize that capability

6. setup new ability to shutdown orteds without sending back an "ack" message to mpirun - not utilized yet, will require some changes to plm terminate_orteds functions in managed environments (coming soon)

Initial tests indicating that fully routing comm via defined routing trees may not actually have a significant cost for operations like IB QP setup. More tests required to confirm.

This will require an autogen...

This commit was SVN r19866.
2008-10-31 21:10:00 +00:00
George Bosilca
0ce76248e8 Close the file descriptors used to push or pull the data to the children.
Without this patch, doing spawn in a loop ended up by exhausting all
available file descriptors pretty quickly. There were about 5 file
descriptors opened per spawned process. Now the number of file
descriptors managed by the process (orted or HNP)
is a lot smaller.

This commit was SVN r19864.
2008-10-31 18:05:28 +00:00
Ralph Castain
30b3bc6761 Minor update - provide one more helpful hint regarding stdin target out-of-range, ensure we exit cleanly since daemons won't have been launched.
This commit was SVN r19847.
2008-10-29 16:00:48 +00:00
Ralph Castain
82ece176d5 Sanity check needs to allow vpid_invalid as this indicates the "none" scenario
This commit was SVN r19820.
2008-10-28 14:50:26 +00:00
Ethan Mallove
2457df91b3 Add missing #include <errno.h> line (for SunStudio Solaris).
This commit was SVN r19814.
2008-10-27 17:41:33 +00:00
Jeff Squyres
c078ab6b09 Minor fix for a trivial compiler warning.
This commit was SVN r19809.
2008-10-27 14:18:49 +00:00
Ralph Castain
71dcf61f9b Add sanity check to ensure that specified stdin target is within range of job. Print error message and exit if not.
Modify read_write test to allow specification of rank to read stdin.

IOF now validated to work for arbitrary rank as stdin target. Not validate to work for multiple simultaneous ranks reading stdin (untested).

This commit was SVN r19804.
2008-10-25 14:38:06 +00:00
Ralph Castain
c56cdac379 Finish cleanup of stdin. Set non-stdio file descriptors to non-blocking (thanks to Jeff for catching that one). Handle writes that result in "would have blocked" errno.
This commit was SVN r19793.
2008-10-24 01:42:58 +00:00
Ralph Castain
6100d88ded Cleanup the new IOF:
1. remove some stale files that were overlooked in original commit

2. add a test program and data to stress iof for stdin

3. cleanup a debug statement that caused memory corruption when reading large files

4. some minor cleanups to correctly handle xon/xoff scenarios

This commit was SVN r19792.
2008-10-23 19:11:05 +00:00
George Bosilca
61317cb61d Complete the r19767 commit for XGrid, i.e. allow the PLM Xgrid to build.
This commit was SVN r19777.

The following SVN revision numbers were found above:
  r19767 --> open-mpi/ompi@6e5d844c36
2008-10-21 15:37:22 +00:00
Ralph Castain
ebaa2c59bb Cleanup non-debug builds
This commit was SVN r19771.
2008-10-18 13:09:47 +00:00
Jeff Squyres
6d026b86b7 Fix a problem reported on the user list by Teng Lin: OPAL_PREFIX
wasn't exported in the Bourne-shell-flavor case on remote nodes.

This commit was SVN r19770.
2008-10-18 12:13:10 +00:00
Jeff Squyres
d96003fec5 Fix typo.
This commit was SVN r19769.
2008-10-18 11:52:41 +00:00
Jeff Squyres
8ea27c0ced Add a missing header file to the Makefile.am so that it can be
included in the distribution tarball.

This commit was SVN r19768.
2008-10-18 11:09:57 +00:00
Ralph Castain
6e5d844c36 Roll in the revamped IOF subsystem. Per the devel mailing list email, this is a complete rewrite of the iof framework designed to simplify the code for maintainability, and to support features we had planned to do, but were too difficult to implement in the old code. Specifically, the new code:
1. completely and cleanly separates responsibilities between the HNP, orted, and tool components.

2. removes all wireup messaging during launch and shutdown.

3. maintains flow control for stdin to avoid large-scale consumption of memory by orteds when large input files are forwarded. This is done using an xon/xoff protocol.

4. enables specification of stdin recipients on the mpirun cmd line. Allowed options include rank, "all", or "none". Default is rank 0.

5. creates a new MPI_Info key "ompi_stdin_target" that supports the above options for child jobs. Default is "none".

6. adds a new tool "orte-iof" that can connect to a running mpirun and display the output. Cmd line options allow selection of any combination of stdout, stderr, and stddiag. Default is stdout.

7. adds a new mpirun and orte-iof cmd line option "tag-output" that will tag each line of output with process name and stream ident. For example, "[1,0]<stdout>this is output"

This is not intended for the 1.3 release as it is a major change requiring considerable soak time.

This commit was SVN r19767.
2008-10-18 00:00:49 +00:00
Jeff Squyres
e34c93c46a Fix problem of missing ) noted by Mostyn Lewis.
This commit was SVN r19758.
2008-10-17 16:03:17 +00:00
Josh Hursey
88aa45dd52 Commit to bring online OpenIB, MX, and shared memory support for Open MPI's checkpoint/restart functionality. Some tuning is still needed, but basic functionality is in place.
There is still a problem with OpenIB and threads (external to C/R functionality). It has been reported in Ticket #1539

Additionally:
* Fix a file cleanup bug in CRS Base.
* Fix a possible deadlock in the TCP ft_event function
* Add a mca_base_param_deregister() function to MCA base
* Add whole process checkpoint timers
* Add support for BTL: OpenIB, MX,  Shared Memory
* Add support Mpool: rdma, sm
* Sundry bounds checking an cleanup in some scattered functions

This commit was SVN r19756.
2008-10-16 15:09:00 +00:00
Ralph Castain
48c3de1865 Fix a problem in the plm "failed to start" code observed by Jeff. When we are unable to launch to a specific node because it doesn't exist or is down, the system would hang and/or segv. The reason for the hang was that we were "firing" the orted exit trigger prior to its timer event being defined - thus "locking" that one-shot and preventing it from firing when we actually were ready to use it.
The segv was caused by the fact that we don't really know which daemon failed to start (at least, in most cases), so we didn't set a pointer to the aborted proc object. All we really wanted, though, was to ensure that mpirun returned a non-zero exit status, so the fix was to simply return the default error status.

This commit was SVN r19754.
2008-10-16 14:21:37 +00:00
Shiqing Fan
3d4e89a5cd - Remove the unused code introduced with r19480, which was for serializing tcp events on Windows and not successful.
This commit was SVN r19747.

The following SVN revisions from the original message are invalid or
inconsistent and therefore were not cross-referenced:
  r19480
2008-10-15 08:39:30 +00:00
Shiqing Fan
8b60c755c2 - Bring r19742 into trunk.
- Unify the Windows and the others way of handling callbacks. Thanks to George.
- This will let Windows use the same callbacks as Linux does, which works also.

This commit was SVN r19746.

The following SVN revisions from the original message are invalid or
inconsistent and therefore were not cross-referenced:
  r19742
2008-10-15 08:14:24 +00:00
Ralph Castain
a7afa869af Bring Jeff's changes over from v1.2 that restores the automatic source of .profile for bash and ksh shells.
This commit was SVN r19709.
2008-10-08 14:21:42 +00:00
Ralph Castain
f4f81c7308 Let the HNP only update the routing tree if necessary. Enable some debug output
This commit was SVN r19676.
2008-10-03 13:41:08 +00:00
Ralph Castain
0cc2e724f8 Separate var declaration from use to remove compiler warnings in non-debug builds
This commit was SVN r19675.
2008-10-03 13:40:31 +00:00
Ralph Castain
15c47a2473 Revise the daemon collective system to handle comm_spawn patterns that cross into new nodes that are not direct children on the routing tree of the HNP.
Refers to ticket #1548. Although this appears to fix the problem, the ticket will be held open pending further test prior to transition to the 1.3 branch.

This commit was SVN r19674.
2008-10-02 20:08:27 +00:00
Ralph Castain
aa11e0977c Correct a bug in the bookmarking code that incorrectly looked at #slots instead of #slots_allocated, thus causing slot reductions in hostfiles to be ignored when selecting our starting node.
Fixes trac:1527

This commit was SVN r19656.

The following Trac tickets were found above:
  Ticket 1527 --> https://svn.open-mpi.org/trac/ompi/ticket/1527
2008-09-29 14:09:02 +00:00
Ralph Castain
4f89adae0c Prettify the user level display of allocation and map to make it easier to see and understand
This commit was SVN r19655.
2008-09-28 16:44:09 +00:00
Ralph Castain
508cb45583 Add a little more diagnostic info when we cannot do an rml send
This commit was SVN r19654.
2008-09-28 02:13:49 +00:00
Ralph Castain
edb3d99687 Update SLURM environmental variables used to describe allocation. Retain backwards compatibility to SLURM 1.1 and earlier versions.
This commit was SVN r19647.
2008-09-26 02:38:37 +00:00
Kenneth Matney
91bbc6b919 Change algorithm from spawning a shell that spawns another shell, and
thereby runs apstat twice; and in the process thereof reads the ALPS
appinfo file TWICE; and in addition, experiences a failure sometimes
which causes mpirun to hang.  Change this to a looped read attempt
that breaks on success, thereby avoiding failure (except in the most

This commit was SVN r19642.
2008-09-25 20:44:16 +00:00
Ralph Castain
037231fbcb MOdify the node_rank and local_rank fields to be uint16_t so we can handle more than 256 procs/node. Change the type to a defined one so that any future change can be easily done, if required.
This commit was SVN r19637.
2008-09-25 13:39:08 +00:00
Ralph Castain
e64b79f30f Modify the --display-map and --display-alloc per note on devel list to reduce info for user understanding.
Add --display-devel-map and --display-devel-alloc to display all the detailed info we used to provide - it is only of use/interest to developers anyway and confuses users.

This commit was SVN r19608.
2008-09-23 15:46:34 +00:00
Josh Hursey
0cd65bfaa8 Fix a SIGPIPE that may occur when checkpointing a restarted process. This was a result of calling system() in the BLCR CRS. After inspection and testing it was determined that the operation was no longer necessary. So the call was removed thus fixing the bug.
This commit was SVN r19601.
2008-09-22 16:49:56 +00:00
Ralph Castain
16e4b0b698 Ensure that a child job inherits its parent job's prefix dir during comm_spawn operations
This commit was SVN r19538.
2008-09-10 19:05:23 +00:00
Ralph Castain
f326ee356e Add some error output to the plm rsh
This commit was SVN r19532.
2008-09-10 01:59:49 +00:00
Ralph Castain
c0d7fbaf88 A few mapping cleanups - mostly aimed to properly balancing loads so multi app-context comm_spawns don't dump everything on one node.
This commit was SVN r19519.
2008-09-08 15:45:55 +00:00
Ralph Castain
9b8473fdbf Cleanup orted cmd line - we don't need to pass nodenames, and shouldn't pass heartbeat unless the orted is going to use it. This helps shorten the cmd line for future use.
Cleanup when an orted actually opens the PLM. Unfortunately, some unmentionable people are pushing head node environs out to remote nodes, causing the daemons to think they are the HNP. This helps prevent the confusion.

This commit was SVN r19518.
2008-09-08 15:45:11 +00:00
Shiqing Fan
04ee20a880 - Mainly type casts. Microsoft VC++ compiler is too strict.
This commit was SVN r19517.
2008-09-08 15:39:30 +00:00
Josh Hursey
edf52e7258 This commit should fix and close #1482
The problem was that (outside of Odin configure issues) that the IOF is no longer enabled by application processes.

Checkpoint/restart seems to be working once again.

Thanks to Ralph for pointing me here.

This commit was SVN r19508.
2008-09-05 18:39:17 +00:00
George Bosilca
579d70edad We should use #ifdef and not #if
This commit was SVN r19504.
2008-09-05 12:44:19 +00:00
Brian Barrett
79cf946bce Add header file needed for the non-full RTE, non-debug case
This commit was SVN r19475.
2008-09-01 18:02:32 +00:00
Shiqing Fan
cd6ff74d89 Update the ccp module:
rename the get_cluster_message function for both ras/plm.
  use _umask instead of umask.
  add WIN32_DCOM definition to support Windows Vista.

This commit was SVN r19470.
2008-09-01 16:35:38 +00:00
Brian Barrett
a15171e46b Some fixes for the disabled ORTE case
* Protect an orte variable used in the orte debugger stuff
  * Initialize the datatype code in the Catamount code, as we need it
    for intercommunicators (the proc code needs it to pack the remote
    name)
  * Turn on a bunch of the orte datatype code so that ORTE_NAME is available.

This commit was SVN r19457.
2008-08-31 18:06:55 +00:00
Shiqing Fan
94a2147e3d - make sure that the system has the header files.
This commit was SVN r19400.
2008-08-25 13:56:10 +00:00
Ralph Castain
b45029fd0e Application processes should not open/close the IOF framework - there is nothing in that framework for application procs to do.
Fix a bug in iof_base_close where we destruct a thread lock prior to unlocking it.

This commit was SVN r19392.
2008-08-22 01:28:19 +00:00
Ralph Castain
4ef9d15d97 Revamp the opal mca paffinity interface. We ran into a problem when we encountered machines that had "holes" in their physical processor layout - e.g., machines that supported "hotplugging", or that had unpopulated sockets. To solve that problem, we had to clarify at the API level where we were describing physical vs logical processor info, and then translate accordingly in the underlying implementation.
See opal/mca/paffinity/paffinity.h for explanation as to the physical vs logical nature of the params used in the API.

Fixes trac:1435

This commit was SVN r19391.

The following Trac tickets were found above:
  Ticket 1435 --> https://svn.open-mpi.org/trac/ompi/ticket/1435
2008-08-21 19:21:28 +00:00
Ralph Castain
43f8bcfe54 Update slurm plm to respect leave_session_attached
This commit was SVN r19370.
2008-08-19 18:30:30 +00:00
Ralph Castain
4e0f34a062 When we hit an error prior to actually launching daemons, it would be nice if orterun didn't bark about daemons failing to launch, mpirun detecting a job failed, etc.
Add a new job state to indicate that we never attempted to launch. Flag such a scenario and avoid hitting all the other error messages.

This commit was SVN r19366.
2008-08-19 15:19:30 +00:00
Ralph Castain
9447334749 Some comments relating to relative indexing
This commit was SVN r19365.
2008-08-19 15:17:40 +00:00
Ralph Castain
49745c5f40 Provide a new option that allows a user to leave an ssh session open without getting deluged by ORTE debug output. The new option is --leave-session-attached, with a corresponding MCA param of orte_leave_session_attached.
Theoretically, any PLM could use this - but in reality, all of them except rsh/ssh already leave the session attached anyway.

This fixes trac:656 - a REALLY old ticket

This commit was SVN r19294.

The following Trac tickets were found above:
  Ticket 656 --> https://svn.open-mpi.org/trac/ompi/ticket/656
2008-08-14 18:59:01 +00:00
Ralph Castain
dd16e4e4a6 Update process component of odls.
This commit was SVN r19281.
2008-08-13 20:07:59 +00:00
Ralph Castain
913cf04633 Only co-locate debugger daemon if the orted has local children - prevents mpirun from co-locating a daemon when it has no local procs
This commit was SVN r19280.
2008-08-13 20:06:28 +00:00
Ralph Castain
3e2a3db887 Add a missing ntoh conversion when pushing a message back onto the RML progress queue.
If a message cannot be routed because the addressee isn't yet known, then the message is held on a queue in the RML for a period of time (currently set to 500 millisec). At the end of that time, we pop the message from the list and attempt to send it again. This action requires that we convert the header back to
network-byte-order before calling the OOB.

If the message still cannot be routed, we put the message back on the list and reset the timer. However, since we are going to convert the header when it com
es off of the list, we have to ntoh it before putting it back on the list so it all comes out right. This step was missing.

Thus, the problem only showed up relatively rarely because a message would have to be pushed onto the queue at least twice for the problem to surface.

This should fix a specific ticket (1389), but we will wait to see the results of MTT runs to verify. Note that we really don't know why a message is rattling around in the RML for so long, especially since this all seems to be happening during finalize, so this could cause mpirun to hang. Or it could simply trash the message and exit cleanly. Shall be interesting to see!

This commit was SVN r19276.
2008-08-13 17:54:15 +00:00
Ralph Castain
30f37f762d Enable co-location of debugger daemons during initial launch and when debugging a running job.
Provide support for four MPIR extensions that allow specification of debugger daemon executable, argv for the debugger daemon, whether or not to forward debugger daemon IO, and whether or not debugger daemon will piggy-back on ORTE OOB network. Last is not yet implemented.

No change in behavior or operation occurs unless (a) the debugger specifically utilizes the extensions and, for co-locate while running, the user specifically enables the capability via an MCA param. Two of the MPIR extensions supported here are used in a widely-used debugger for a large-scale installation. The other two extensions are new and being utilized in prototype work by several debuggers for possible future release.

This commit was SVN r19275.
2008-08-13 17:47:24 +00:00
Rainer Keller
d57ef70149 - Store the result of the 1-byte read... and assert, in case
of error checking -- we don't return errors here anyway.
   Fixes Coverity CID 981

This commit was SVN r19259.
2008-08-12 18:00:38 +00:00
Ralph Castain
f017c55bfa Close a minor memory leak - we can reuse timer events
This commit was SVN r19251.
2008-08-12 12:53:30 +00:00
Ralph Castain
baed5dcad0 Ensure contact info is placed in the job family session directory so orte-ps and other tools can find it
This commit was SVN r19245.
2008-08-11 23:48:39 +00:00
Rainer Keller
0f8a80d81d - The intel compiler does not play nice with the
__opal_attribute_format__ on typedef defined functions and
   emits a warning once errmgr.h is included --> read: often...

This commit was SVN r19229.
2008-08-08 16:26:09 +00:00
Jeff Squyres
797ec531aa Some more work on the man pages:
* Make the creation of the build dir for the man pages a bit more
   robust (thanks to suggestions from Ralf W.).
 * Only distribute the .Xin files, not the .X man pages themselves.
 * Make the .X files depend on opal_config.h so that if you re-run
   configure and change opal_config.h (e.g., a new version), the man
   pages should get rebuilt.
 * Man pages are now cleaned with "distclean", not "maintainer-clean".
 * Fix a typo in opal_crs.7in.
 * Udpate make_dist_tarball to update "date" in the VERSION file.
 * Make make_dist_tarball a bit friendlier to hg checkouts.

This commit was SVN r19219.
2008-08-07 19:20:40 +00:00
Rainer Keller
ad3538ea38 - Per Discussion w/ Ralph Castain (related to CID 1051)
- Move up the __opal_attribute_noreturn__ information
 - Actually make it known outside in ess.h
 - Additionally allow printf-type checking

This commit was SVN r19210.
2008-08-07 09:36:10 +00:00
Ralph Castain
c9e53fd0d4 Add capability to notify system admins of potential problems in system communication networks and/or other system elements that are detected by Open MPI during operation. For example, failures in connections that may be indicative of connectivity problems can be reported to sys admins in addition to our current error message to the user, thus allowing more rapid correction of the problem.
This system is "off" by default and only operates upon specific directive for selection of a notifier component. At the moment, the only available component will write an error message to the syslog.

This commit was SVN r19209.
2008-08-06 21:59:21 +00:00
Rainer Keller
c58e89e471 - Fix variable set but not used
Coverity CID1061

This commit was SVN r19197.
2008-08-06 14:42:16 +00:00
George Bosilca
d8fe05264b Fix recursion in include files (Coverty fix 156).
This commit was SVN r19181.
2008-08-06 13:50:01 +00:00
Ralph Castain
be02211b4f Modify the wakeup system to make it more Windows-friendly. This allows Shiqing to consolidate the Windows-specific modifications into one location, and generalizes the wakeup procedure in case we hit other system-specific requirements.
This needs some soak time to ensure we haven't opened any race conditions. I tried to loop everything in the shutdown procedure through that trigger event call to ensure it all goes through the one-time locks as it did before so that someone hitting ctrl-c when we are already shutting down shouldn't cause problems. Just want to let people use it for awhile to verify.

This commit was SVN r19159.
2008-08-05 15:09:29 +00:00
Ralph Castain
35a86b3347 Establish an MCA param "orte_allocation_required" so that a system can require the user have an RM-provided allocation in order to run. This helps prevent the problem where a user forgets to get an allocation on an RM-managed cluster, and then executes mpirun on the head node - thus causing all of their mpi procs to launch on the head node, usually bringing it to its knees.
Since OMPI allows mpirun to default to the local node, and since users want to retain the option to co-locate procs with mpirun, we needed another param to block this error case.

This commit was SVN r19135.
2008-08-04 14:25:19 +00:00
Rainer Keller
0d08866786 - Declare functions in lex-files as extern "C" {} to get
rid of warnings.

This commit was SVN r19132.
2008-08-04 11:49:01 +00:00
Ralph Castain
5b2f53a069 One more quick fix - ensure we are looking at the value and not its pointer
This commit was SVN r19123.
2008-08-01 23:39:55 +00:00
Jeff Squyres
26c7daf16a Fix typo
This commit was SVN r19121.
2008-08-01 21:30:53 +00:00
Dan Lacher
9175da1e02 Putback for all changes to automate man page updates to strings of
versions, dates and build names.

Fixes trac:1387

Big thanks to Jeff and Brian for help and oversight.

This commit was SVN r19120.

The following Trac tickets were found above:
  Ticket 1387 --> https://svn.open-mpi.org/trac/ompi/ticket/1387
2008-08-01 21:14:37 +00:00
Ralph Castain
21cd4b9df8 Add pls_rsh_agent synonym to the PLM rsh component
This commit was SVN r19119.
2008-08-01 20:15:42 +00:00
Jeff Squyres
5818eca234 Also make sure that the new INTERNAL channel doesn't close the
endpoint and/or the real stderr fd in the HNP.

This commit was SVN r19113.
2008-07-31 21:26:58 +00:00
Ralph Castain
2ee493c3f9 Fix some FT code to reflect change in session_dir interface
This commit was SVN r19106.
2008-07-31 14:53:18 +00:00
Ralph Castain
a62b2a0150 Per the July technical meeting:
Standardize the handling of the orte launch agent option across PLMs. This has been a consistent complaint I have received - each PLM would register its own MCA param to get input on the launch agent for remote nodes (in fact, one or two didn't, but most did). This would then get handled in various and contradictory ways.

Some PLMs would accept only a one-word input. Others accepted multi-word args such as "valgrind orted", but then some would error by putting any prefix specified on the cmd line in front of the incorrect argument.

For example, while using the rsh launcher, if you specified "valgrind orted" as your launch agent and had "--prefix foo" on you cmd line, you would attempt to execute "ssh foo/valgrind orted" - which obviously wouldn't work.

This was all -very- confusing to users, who had to know which PLM was being used so they could even set the right mca param in the first place! And since we don't warn about non-recognized or non-used mca params, half of the time they would wind up not doing what they thought they were telling us to do.

To solve this problem, we did the following:

1. removed all mca params from the individual plms for the launch agent

2. added a new mca param "orte_launch_agent" for this purpose. To further simplify for users, this comes with a new cmd line option "--launch-agent" that can take a multi-word string argument. The value of the param defaults to "orted".

3. added a PLM base function that processes the orte_launch_agent value and adds the contents to a provided argv array. This can subsequently be harvested at-will to handle multi-word values

4. modified the PLMs to use this new function. All the PLMs except for the rsh PLM required very minor change - just called the function and moved on. The rsh PLM required much larger changes as - because of the rsh/ssh cmd line limitations - we had to correctly prepend any provided prefix to the correct argv entry.

5. added a new opal_argv_join_range function that allows the caller to "join" argv entries between two specified indices

Please let me know of any problems. I tried to make this as clean as possible, but cannot compile all PLMs to ensure all is correct.

This commit was SVN r19097.
2008-07-30 18:26:24 +00:00
Ralph Castain
01a7259a7d This fixes ticket #1426 - mpirun is cleaning up ALL session dirs
Mpirun - and the orteds - were doing their best to whack all session dirs on their nodes just in case there was something lingering due to an abnormal termination. Unfortunately, they were -too- good at it. They were whacking all session directories under the user's name, even those from other mpiruns!

This adds another layer to the session dir tree so that we can denote which jobs come from our own job family, and restricts the cleanup operation to only session dirs from within our own job family. So we'll still cleanup anything due to our own mpirun, but won't whack any other mpirun from this user.

Call it being polite...

This commit was SVN r19083.
2008-07-29 18:58:35 +00:00
George Bosilca
a4d905db4a Allow xgrid to compile.
This commit was SVN r19076.
2008-07-29 13:24:08 +00:00
Ralph Castain
1210a96d82 Ensure a value gets defined before used...thanks Jeff
This commit was SVN r19075.
2008-07-29 13:08:45 +00:00
Jeff Squyres
0af7ac53f2 Fixes trac:1392, #1400
* add "register" function to mca_base_component_t
   * converted coll:basic and paffinity:linux and paffinity:solaris to
     use this function
   * we'll convert the rest over time (I'll file a ticket once all
     this is committed)
 * add 32 bytes of "reserved" space to the end of mca_base_component_t
   and mca_base_component_data_2_0_0_t to make future upgrades
   [slightly] easier
   * new mca_base_component_t size: 196 bytes
   * new mca_base_component_data_2_0_0_t size: 36 bytes
 * MCA base version bumped to v2.0
   * '''We now refuse to load components that are not MCA v2.0.x'''
 * all MCA frameworks versions bumped to v2.0
 * be a little more explicit about version numbers in the MCA base
   * add big comment in mca.h about versioning philosophy

This commit was SVN r19073.

The following Trac tickets were found above:
  Ticket 1392 --> https://svn.open-mpi.org/trac/ompi/ticket/1392
2008-07-28 22:40:57 +00:00
Ralph Castain
1a77b15523 Modify the handling of hostfiles to allow them to subdivide allocations. Utilize the "slots_alloc" field of the orte_node_t object - which had previously been unused - to track the #slots allocated to a given app_context. Let the hostfile filtering action utilize the #slots field to modify the allocated slots for each app_context.
This commit was SVN r19066.
2008-07-28 15:10:40 +00:00
Ralph Castain
3107545709 Ensure that ORTE processes such as mpirun and orted never inadvertently bind themselves to cores. Change the mca param name used by the rank_file mapper to get user directives on slot lists to be different from that used by MPI procs to discover their binding. Add a cmd line option to orterun to make it easier for a user to specify the slot list (basically, hide the mca param name).
Discussed and reviewed with Lenny and Jeff.

This commit was SVN r19062.
2008-07-28 14:18:36 +00:00
Ralph Castain
0735d6f1c2 This commit fixes ticket #1414
Cleanup the logic in the odls for when processes terminate. It turns out that we were only going through the kill_proc logic once instead of looping over all local children when we ordered a daemon to kill its local procs. This went unnoticed for some time as for most systems the local procs were terminated anyway when the daemon terminated due to the parent/child relationship.

Solaris is apparently different - the children are not automatically terminated when the parent dies. As a result, it acts as a detector for this bug.

Mucho thanks to Rolf V. for his help in debugging - and to IM for letting me follow his gdb progress in quasi real-time!

This commit was SVN r19044.
2008-07-26 02:54:43 +00:00
Ralph Castain
d5a916d350 Fix a problem reported by IBM: nolocal and bynode combined to map byslot. Problem actually was that any time multiple mapping policy directives were provided, we would only map byslot due to incorrect if statement conditions.
Thanks to Kris Davis for his patience while we tracked this down!

This commit was SVN r19039.
2008-07-25 17:50:46 +00:00
Ralph Castain
718cceddaa Ensure that we only launch procs on the HNP if that node is actually included in the allocation.
This commit was SVN r19038.
2008-07-25 17:13:22 +00:00
Ralph Castain
cb93775cca Just for the AR - remove unnecessary typecast
This commit was SVN r19034.
2008-07-25 15:30:37 +00:00
Thomas Herault
28dc80b67e Deal with the SIGCHLD issue in LSF.
lsb_launch tampers with SIGCHLD signal handler. We are forced to reinstall our own signal handler after a call to this function.

This commit fixes trac:1356.

This commit was SVN r19033.

The following Trac tickets were found above:
  Ticket 1356 --> https://svn.open-mpi.org/trac/ompi/ticket/1356
2008-07-25 15:23:23 +00:00
Ralph Castain
7e6e104fc3 Add more debugging to the RML when it fails to find a route - specifically, have it print a stacktrace so we can figure out where it came from.
This commit was SVN r19032.
2008-07-25 15:01:41 +00:00
Ralph Castain
42c134cb32 Silence stupid compiler warning - and a certain someone who keeps reminding me of it... :-)
This commit was SVN r19031.
2008-07-25 14:01:06 +00:00
Ralph Castain
a1d296ae03 This commit fixes ticket #1410
Fix a few bugs in the mappers:

1. Ensure that bynode with no -np fills all available slots - it just does so with the ranks set bynode instead of byslot

2. fix --nolocal behavior so it works correctly in all cases. We still have to test the host's name using opal_ifislocal in the mapper because the name returned by gethostname to orte_process_info.hostname can be an FQDN, but a hostfile may contain a non-FQDN version.

3. Add missing --nolocal logic to the seq mapper

Oversubscribed mapping seemed to be working okay without repair, so I couldn't verify my own bug report in that regard.

Also included are some preliminary changes to support the modified hostfile behavior, which will be committed shortly:

1. removed the totally useless "allocate" field in the orte_node_t object since every node is automatically allocated for use - and everything ignored the field anyway

2. correctly initialize the slots_alloc field when the allocation is read

This commit was SVN r19030.
2008-07-25 13:35:12 +00:00
Ralph Castain
fdb2408bf2 Rename the osx paffinity component the "posix" component since it really has nothing osx specific in it - it is just a generic posix call to determine #processors. Set the priority low so that both linux and solaris components override it if they build. It shouldn't build in Windows at all.
Modify the odls to remove a (size_t) typecast in front of the num_processors variable just in case it is returned negative. This usually is accompanied by an opal_error, so this shouldn't make any difference - but it is more technically correct.

This commit was SVN r19008.
2008-07-24 01:54:51 +00:00
Lenny Verkhovsky
b4d54dda57 Fixed possible seqf when using RANKFILE, but not all ranks assigned
Fixed allocation of all ranks when using RANKFILE, but not all ranks assigned
Aborting if using RANKFILE, but np wasn't specified a little earlier
Clean mca_rmaps_rank_file_component.debug

This commit was SVN r19004.
2008-07-23 17:44:02 +00:00
Ralph Castain
e3c3d28bf1 Add some more debugging to tell us how many processors were found when setting sched_yield
This commit was SVN r18999.
2008-07-23 15:28:51 +00:00
Thomas Herault
b6affd35e9 Small typos for LSF compilation and update Makefile.am
This commit was SVN r18998.
2008-07-23 14:42:26 +00:00
Ralph Castain
83e7c19d33 Remove deprecated function - this was incorporated into the paffinity framework a long time ago. Fortunately, nobody was actually using it!
This commit was SVN r18990.
2008-07-23 03:43:31 +00:00
Ralph Castain
76f2659527 Very minor cleanup to slurm support
This commit was SVN r18987.
2008-07-23 02:35:03 +00:00
Ralph Castain
1f665425e7 Fix some compile problems in the LSF support
This commit was SVN r18986.
2008-07-23 02:34:41 +00:00
Ralph Castain
a4f0fa6e3a Update the routed framework to:
1. add a new API delete_route(orte_process_name_t*) to delete the specified proc from the routing table

2. modify update_route so that it actually updates pre-existing routes instead of only adding routing info the end of the hash table

This fixes ticket #1403

This commit was SVN r18970.
2008-07-21 21:37:09 +00:00
Jeff Squyres
54dbd95243 Fix some component version numbers to be the same as the OMPI release
This commit was SVN r18965.
2008-07-21 20:05:29 +00:00
George Bosilca
bcac9a0540 Remove a warning about using map when it is not initialized.
This commit was SVN r18957.
2008-07-21 14:35:05 +00:00
Jeff Squyres
750ea30961 So apparently my clever fix in r18873 was not good -- apparently, we
can have a pub_endpoint and a sub_endpoint that are not equal but go
to the same place (fd).  I didn't think that that was possible. :-\

So just use a bool to track whether we have forwarded the fragment at
all; if we have, then don't forward to the sub_endpoint.

IOF is going to be re-written for v1.4.

This commit was SVN r18950.

The following SVN revision numbers were found above:
  r18873 --> open-mpi/ompi@773c92a6eb
2008-07-18 20:04:26 +00:00
Ralph Castain
6135943382 Update the paffinity call in the ODLS so we retrieve the number of processors on the local node, thus allowing us to correctly set the sched_yield parameter.
This commit was SVN r18946.
2008-07-18 19:19:16 +00:00
Ralph Castain
09db4c3a60 Add target tag to diagnostic output
This commit was SVN r18918.
2008-07-16 01:57:01 +00:00
Ralph Castain
07841808ee Add some debugging - provide a gentler abort when routes cannot be found
This commit was SVN r18915.
2008-07-15 15:48:46 +00:00
Lenny Verkhovsky
a812324963 Fixing "paffinity_base_slot_list" environment
This commit was SVN r18900.
2008-07-14 07:10:50 +00:00
Ralph Castain
b1f367563c Few minor code cleanups
This commit was SVN r18890.
2008-07-11 15:40:41 +00:00
Ralph Castain
58964b2bf8 Make lsf support compile
This commit was SVN r18889.
2008-07-11 15:40:25 +00:00
Jeff Squyres
583bf425c0 Fixes trac:1383:
Short version: remove opal_paffinity_alone and restore
mpi_paffinity_alone.  ORTE makes various information available for the
MPI layer to decide what it wants to do in terms of processor
affinity.

Details:

 * remove opal_paffinity_alone MCA param; restore mpi_paffinity_alone
   MCA param
 * move opal_paffinity_slot_list param registration to paffinity base
 * ompi_mpi_init() calls opal_paffinity_base_slot_list_set(); if that
   succeeds use that.  If no slot list was set, see if
   mpi_paffinity_alone was set.  If so, bind this process to its Node
   Local Rank (NLR).  The NLR is the ORTE-maintained slot ID; if you
   COMM_SPAWN to a host in this ORTE universe that already has procs
   on it, the NLR for the new job will start at N (not 0).  So this is
   slightly better than mpi_paffinity_alone in the v1.2 series.
 * If a slot list is specified *and* mpi_paffinity_alone is set, we
   display an error and abort.
 * Remove calls from rmaps/rank_file component to register and lookup
   opal_paffinity mca params. 
 * Remove code in orte/odls that set affinities - instead, have them
   just pass a slot_list if it exists. 
 * Cleanup the orte/odls code that determined
   oversubscribed/want_processor as these were just opposites of each
   other.

This commit was SVN r18874.

The following Trac tickets were found above:
  Ticket 1383 --> https://svn.open-mpi.org/trac/ompi/ticket/1383
2008-07-10 21:12:45 +00:00