Ralph Castain
70bef5ca95
It is possible for the jdata param to be NULL, so protect the verbose output in that case
...
This commit was SVN r23109.
2010-05-07 16:48:44 +00:00
Shiqing Fan
8ea8563462
Include the new component config file for Windows into tarball.
...
This commit was SVN r23108.
2010-05-07 14:30:12 +00:00
Ralph Castain
d6a1d7a082
Little more cleanup on paffinity. Provide a specific error code for affinity not supported so we can better report the problem. Move the error reporting to orterun so we only get one error message. Update the darwin paffinity module to return the correct new error codes.
...
This commit was SVN r23107.
2010-05-07 14:04:55 +00:00
Ralph Castain
d4f56cff61
More cleanup on paffinity....groan
...
It is okay to not have a paffinity module IF you aren't using paffinity anyway. So don't error out of MPI_Init because a paffinity module wasn't selected.
Cleanup error reporting in the odls default module to (once and for all!) eliminate messages originating in the fork'd process. Create some new error codes to allow us to pass enough info back to the parent process to provide useful error messages.
This commit was SVN r23106.
2010-05-06 20:57:17 +00:00
Jeff Squyres
477201e161
Fix "make dist" breakage
...
This commit was SVN r23105.
2010-05-06 18:47:20 +00:00
Shiqing Fan
76726f6094
Update a few module configuration for Windows.
...
This commit was SVN r23104.
2010-05-05 12:22:04 +00:00
Ralph Castain
2ff1ae13e1
Create a new "heartbeat" module in the sensor framework and move the plm_base heartbeat code there. Add new proc and job states for heartbeat_failed. Remove the "heartbeat" cmd line option for orted as this is now done automatically if the --enable-heartbeat configure option is set.
...
This commit was SVN r23102.
2010-05-05 00:48:43 +00:00
Ralph Castain
99f223210d
Add some contributed examples of how to start and configure the spread library. Do a little more cleanup on the spread module, and ensure that it isn't selected if spread isn't running.
...
This commit was SVN r23101.
2010-05-04 23:44:00 +00:00
Ralph Castain
43005b88e0
Catch one more spot...thanks to Sam for reporting it
...
This commit was SVN r23094.
2010-05-04 17:21:15 +00:00
Ralph Castain
f4ae2885e2
Add new error constant
...
This commit was SVN r23090.
2010-05-04 13:44:33 +00:00
Ralph Castain
cd569f8a79
Restore the global restart capability
...
This commit was SVN r23089.
2010-05-04 02:40:29 +00:00
Ralph Castain
3ca0b4138b
Let the nidmap functions update a new orte_process_info field as to the number of daemons in the system
...
This commit was SVN r23088.
2010-05-04 02:40:09 +00:00
Ralph Castain
8c5f442ee0
Fix some bugs in the spread rmcast component
...
This commit was SVN r23086.
2010-05-04 02:38:37 +00:00
Ralph Castain
8e7faf9119
Add a new test for the db framework, fix some minor bugs in the daemon module
...
This commit was SVN r23085.
2010-05-04 02:38:11 +00:00
Ralph Castain
9dfb5c7c62
Rename the orte state framework to be "db", which more accurately reflects its overall capabilities since it can store any kind of data (not just state, although that will be its primary purpose). Update tools and tests accordingly. Add a daemon module for storing data on the daemons - requires --enable-multicast, so it won't build unless that is set
...
This commit was SVN r23082.
2010-05-03 04:11:03 +00:00
Ralph Castain
5103ffead6
Continue work on local recovery for orteds
...
This commit was SVN r23080.
2010-05-03 04:08:13 +00:00
Ralph Castain
bcff0d6301
Some minor cleanup in the rmcast framework, ensure that a default multicast group is always defined for each app
...
This commit was SVN r23079.
2010-05-03 04:07:14 +00:00
Ralph Castain
f994a7edf4
Add recovery data to the jobdat object
...
This commit was SVN r23078.
2010-05-03 04:06:13 +00:00
Ralph Castain
323224b84b
Allow the restart data to be retrieved from a job's app->env or default that given to orterun
...
This commit was SVN r23077.
2010-05-03 04:05:19 +00:00
Ralph Castain
3f262bf0b6
Add a new reliable multicast component based on the "spread" library
...
Thanks to Srini Nariangadu (Cisco) for the contribution!
This commit was SVN r23076.
2010-05-02 17:29:41 +00:00
Ralph Castain
c93af95351
Since there is no defined behavior for the case where all application procs exit normally, but some or all have non-zero returns, just output a warning telling the user how many procs meet that criteria. Let the return code of mpirun in that scenario reflect any errors in OMPI/ORTE itself.
...
Clearly a temporary solution until a defined behavior can be established.
This commit was SVN r23075.
2010-04-30 19:01:10 +00:00
Ralph Castain
c62418d76d
Add a new test that checks behavior when we call exit with a non-zero return code after calling finalize - don't ask why.
...
Modify the check_complete code so it finds the first non-zero exit status (i.e., the one from the lowest rank) in a job that terminates normally, and sets the mpirun exit code to that status.
This commit was SVN r23071.
2010-04-29 19:58:44 +00:00
Ralph Castain
5689bd6d30
Remove debug
...
This commit was SVN r23062.
2010-04-28 22:14:23 +00:00
Ralph Castain
319758e3e0
Restore process recovery for procs local to mpirun (first step towards restoring full capability). Define three new MCA params:
...
1. orte_enable_recovery - default recovery policy, can be overridden on a per-job basis
2. orte_max_local_restarts - default max number of local restarts, can be overridden
3. orte_max_global_restarts - default max number of relocates, can be overridden
Implement the restart_proc API for the ODLS framework, reorganize the default fns a little to avoid copying code.
This commit was SVN r23057.
2010-04-28 04:06:57 +00:00
Ralph Castain
67568e7dec
Fix a warning
...
This commit was SVN r23049.
2010-04-27 15:59:24 +00:00
Ralph Castain
401fb22847
Fix the pack/unpack of process state values by getting the actual packing size to match the new value size.
...
Improve the debug output by printing the string
This commit was SVN r23048.
2010-04-27 15:58:04 +00:00
Ralph Castain
efc7d3e2f6
Update the LSF PLM module to ensure that the path is correctly passed.
...
Many thanks to Teng Lin for the error report - and the patch!
This commit was SVN r23047.
2010-04-27 03:46:04 +00:00
Ralph Castain
3434296836
Ensure we don't have a trailing separator on the end of our tmpdir as (a) it really looks weird, and (b) some exotic systems interpret that as indicating the rest of the path is to be treated as absolute. Makes for very strange and interesting behavior...
...
This commit was SVN r23046.
2010-04-27 03:40:44 +00:00
Ralph Castain
55889934d8
After hours spent chasing the stupid "abort" file, it became clear that we were always going to be plagued by that idiot contraption when trying to be good citizens and properly cleanup. So get rid of it by instead doing a messaging handshake with the local daemon.
...
Note that this isn't a problem since MPI_Abort and orte_abort are only called under controlled circumstances - i.e., we are doing an orderly abort and not segfaulting. If we can't get the message out for some reason, then too bad - we'll still see an abnormal process termination and act accordingly.
This commit was SVN r23045.
2010-04-27 03:39:32 +00:00
Ralph Castain
b9893aacc5
Add a sensor framework to ORTE that monitors applications and notifies the errmgr when they exceed specified boundaries. Two modules are included here:
...
1. file activity - can monitor file size, access and modification times. If these fail to change over a specified number of sampling iterations (rate is an mca param), then the errmgr is notified.
2. memory usage - checks amount of memory used by a process. Limit and sampling rate can be set.
This support must be enabled by configuring --enable-sensors.
ompi_info and orte-info have been updated to include the new framework.
Also includes some initial steps toward restoring the recovery capability. Most notably, the ODLS API has been extended to include a "restart_proc" entry for restarting a local process, and organizes the various ERRMGR framework globals into a single struct as we do in the other ORTE frameworks. Fix an oversight in the ERRMGR framework where a pointer array was constructed, but not initialized.
Implementation continues.
This commit was SVN r23043.
2010-04-26 22:15:57 +00:00
Ralph Castain
43a89bbace
Extend the process and job states by adding values for exceeding sensor bounds. This changes the job state field to 32-bit to also provide room for future expansion.
...
This commit was SVN r23036.
2010-04-26 12:36:40 +00:00
Ralph Castain
e3164d2ac1
For the autogen-challenged (i.e., Jeff), create a new ORTE constant that tells the user that a required module was not found. Update the errmgr select function to output the error if no module is found.
...
This commit was SVN r23032.
2010-04-24 01:39:26 +00:00
Ralph Castain
b62ebb8b94
Enable these to build so others can more easily begin implementation
...
This commit was SVN r23031.
2010-04-23 22:46:33 +00:00
Ralph Castain
9c69900bbc
Ensure we get the jdata object when updating proc state so we don't segv if report-progress is set. Thanks to Sam for reporting it!
...
This commit was SVN r23029.
2010-04-23 15:14:21 +00:00
George Bosilca
95394456ca
No Windows EOF.
...
This commit was SVN r23024.
2010-04-23 05:51:29 +00:00
Ralph Castain
efbb5c9b7c
Revamp the errmgr framework to provide a greater range of optional behaviors, including different behaviors for daemons, and remove several looping messages across the code base:
...
* add hnp and orted modules to the errmgr framework. The HNP module contains much of the code that was in the errmgr base since that code could only be executed by the HNP anyway.
* update the odls to report process states directly into the active errmgr module, thus removing the need to send messages looped back into the odls cmd processor. Let the active errmgr module decide what to do at various states.
* remove the code to track application state progress from the plm_base_launch_support.c code. Update the plm modules to call the errmgr directly when a launch fails.
* update the plm_base_receive.c code to call the errmgr with state updates from remote daemons
* update the routed modules to reflect that process state is updated in the errmgr
* ensure that the orted's open the errmgr and select their appropriate module
* add new pretty-print utilities to print process and job state. Move the pretty-print of time info to a globally-accessible place
* define a global orte_comm function to send messages from orted's to the HNP so that others can overlay the standard RML methods, if desired.
* update the orterun help output to reflect that the "term w/o sync" error message can result from three, not two, scenarios
This commit was SVN r23023.
2010-04-23 04:44:41 +00:00
Shiqing Fan
d1e66bdd01
Use variables instead of hard-coded compiler flags, in order to support various C/C++ compilers on Windows.
...
This commit was SVN r23016.
2010-04-21 12:45:00 +00:00
Ralph Castain
86228aee38
Provide two new opal paffinity utilities for printing a hex representation of the cpu set and parsing that string back into a cpu set on the other end. Also add a new MCA param for passing the cpu set applied to a process during launch down to that process so it can know what we attempted to do.
...
All to be used in some new MPI extensions provided by Jeff so that users can easily query their binding situation.
This commit was SVN r22998.
2010-04-19 22:16:35 +00:00
Ralph Castain
4d06125a33
Establish a method by which a process knows if it has been bound by mpirun. This helps resolve a problem where a process gets "bound" to all available resources, which looks to the opal paffinity system as "not bound". This can cause mpi_init to attempt to "bind" the process itself, causing unintended behavior.
...
This commit was SVN r22985.
2010-04-17 01:58:26 +00:00
Ralph Castain
41428e6b61
Issue a warning if a requested binding operation results in processes being bound to all available processes, which is the equivalent of not being bound at all.
...
See the following email thread for further details:
http://www.open-mpi.org/community/lists/devel/2010/04/7745.php
This commit was SVN r22984.
2010-04-17 01:02:41 +00:00
Ralph Castain
6c379fed2e
IOF components should not assume they will be selected when queried - thus, they should not perform init functions until after selection. Create init/finalize entry points for that purpose, and have select init the module after it has been selected.
...
This commit was SVN r22982.
2010-04-16 18:51:27 +00:00
Ralph Castain
2ecc9fc2b3
Additional diag output
...
This commit was SVN r22981.
2010-04-16 14:48:37 +00:00
Ralph Castain
4308922f59
Ensure that any application-specific selection of ess module doesn't get overridden by what is given to the orted or orterun
...
Cleanup tool name determination for CM
This commit was SVN r22980.
2010-04-15 18:10:50 +00:00
Ralph Castain
c3e4c40cdf
Add another multicast tag for updating state
...
This commit was SVN r22979.
2010-04-15 18:08:53 +00:00
Ralph Castain
eeccf2f15c
Some minor changes to support vm's
...
This commit was SVN r22975.
2010-04-14 01:20:43 +00:00
Ralph Castain
854dc12fc0
Add a tag for multicasting IOF messages
...
This commit was SVN r22972.
2010-04-13 22:51:26 +00:00
Ralph Castain
81329c637e
Indentation corrections
...
This commit was SVN r22971.
2010-04-13 17:47:34 +00:00
Ralph Castain
8da781af84
Continue developing support for distributed virtual machines - minor changes to ensure correct jobid gets used and that dvm's can communicate with tools
...
This commit was SVN r22958.
2010-04-12 22:33:09 +00:00
Shiqing Fan
96b20a29b5
An easy solution to make singleton work on Windows.
...
This commit was SVN r22952.
2010-04-10 16:30:59 +00:00
Ralph Castain
d3ed4e68b7
Utilize a non-used mapping policy bit to define a policy that uses only existing alive daemons to support virtual machines and restarting processes on already-active nodes
...
This commit was SVN r22951.
2010-04-10 05:02:47 +00:00
Ralph Castain
4f8279df3d
Enable substitution of the communication calls in the orted when sending messages back to the HNP by creating a function for this purpose and saving the pointer to it in orte_odls_base. Higher level libraries can then override the default function to use their own method.
...
This commit was SVN r22950.
2010-04-09 18:50:10 +00:00
Ralph Castain
c32f046d7c
Tiny cleanup - when the user kills us with a ctrl-c, there really isn't a need to tell him "your procs died and we don't know why". Just shaddup and die.
...
This commit was SVN r22949.
2010-04-09 18:47:35 +00:00
Terry Dontje
282a537cf7
This commit fixes 2370, by having the solaris paffinity module return error codes for get_physical_processor_id and having odls_default_fork_local_proc check get_physical_processor_id for OPAL_ERROR
...
This commit was SVN r22948.
2010-04-09 15:10:46 +00:00
Brad Benton
101b896f2e
IBM has approved the release of the LoadLeveler sample code under the
...
BSD license. Consequently, a more restrictive licensing clause that was
originally associated with the LoadLeveler sample code documentation and
replicated in a comment block in this file has been removed.
This commit was SVN r22947.
2010-04-08 19:41:44 +00:00
Terry Dontje
929c58e38d
This commit fixes trac:2073
...
This commit was SVN r22946.
The following Trac tickets were found above:
Ticket 2073 --> https://svn.open-mpi.org/trac/ompi/ticket/2073
2010-04-08 18:17:44 +00:00
Ralph Castain
75e99e6118
Do a better job of selecting cm ess component, handle tool and daemon issues
...
This commit was SVN r22942.
2010-04-07 18:59:21 +00:00
Ralph Castain
f1fc344336
Add some diagnostics
...
This commit was SVN r22941.
2010-04-07 18:58:17 +00:00
Ralph Castain
8e29a6858a
Properly handle the case when a daemon is given both parts of its name
...
This commit was SVN r22935.
2010-04-06 22:41:18 +00:00
Ralph Castain
a1e82e9d05
Per discussion with Josh, cleanup the errmgr API by creating separate modules for the public vs internal APIs. This mirrors the architecture used in other frameworks that had similar requirements.
...
Remove the orcm errmgr module - moving to the orcm code base so it can utilize orcm communications and not interfere with ompi-related operations.
This commit was SVN r22931.
2010-04-05 22:59:21 +00:00
Ralph Castain
1caba7af2f
Fix a bunch of compiler warnings reported by Jeff
...
This commit was SVN r22930.
2010-04-03 00:20:19 +00:00
Ralph Castain
84c7973df8
Update the #procs in the job prior to assigning vpids for each app_context.
...
This commit was SVN r22929.
2010-04-03 00:03:35 +00:00
Ralph Castain
6b43b76f9d
Some updates required for generating a LAM-style virtual machine. Retain the local node if requested. Properly setup the daemon job map for a VM launch.
...
This commit was SVN r22928.
2010-04-03 00:03:01 +00:00
Ralph Castain
de6679dbd3
Truly respect the -quiet option. Make it an mca param so someone doesn't have to put it solely on the cmd line. Tell show_help to shaddup as well.
...
This commit was SVN r22926.
2010-04-02 14:19:38 +00:00
Ralph Castain
25a9a195b0
When the user requests "quiet", mpirun isn't supposed to output "helpful error messages" - so shaddup!
...
This commit was SVN r22925.
2010-04-02 07:13:11 +00:00
Ralph Castain
ed0f42fa49
Fix a bug courtesy of Jeff - since check_job_complete removes the child object and releases it, preserve the pointer to the next item on the list prior to working with it
...
This commit was SVN r22924.
2010-04-02 07:08:34 +00:00
Jeff Squyres
c57a8fba5a
Improve the help message when mpirun cannot find an executable.
...
Refs trac:2035
This commit was SVN r22922.
The following Trac tickets were found above:
Ticket 2035 --> https://svn.open-mpi.org/trac/ompi/ticket/2035
2010-04-01 13:26:29 +00:00
Ralph Castain
871a9e0df4
Track process heartbeats with time_t, be a little less restrictive on who can retrieve an orte_job_t object
...
This commit was SVN r22921.
2010-03-31 19:20:06 +00:00
Josh Hursey
62f8d3c471
r22885 missed a few symbol updates when it changed ompi_want_ft to opal_want_ft
...
This commit was SVN r22916.
The following SVN revision numbers were found above:
r22885 --> open-mpi/ompi@522a23d6a3
2010-03-30 16:47:39 +00:00
Ralph Castain
f6bfaa76ba
Add some debug output to job_complete. If no session dirs were created, then cannot check for abort file - which wouldn't be created anyway
...
This commit was SVN r22903.
2010-03-29 23:21:03 +00:00
Ralph Castain
24c3b4f849
Add the sysinfo framework to the "info" tools, especially since the odls_base_open function calls it!
...
This commit was SVN r22901.
2010-03-29 20:47:29 +00:00
Ralph Castain
2603bd8a47
Eliminate a race condition (first reported by Josh) when deliberately killing procs. Need to cancel the waitpid callback for the proc, then properly flag it as dead (both not-alive and waitpid-fired) so that the system cleans up properly.
...
This commit was SVN r22900.
2010-03-28 16:08:05 +00:00
Ralph Castain
4f9db20d94
Couple of minor cleanups
...
This commit was SVN r22899.
2010-03-28 15:41:27 +00:00
Ralph Castain
1bf9684ebb
Don't include jobs in the nidmap if they aren't mapped jobs
...
This commit was SVN r22886.
2010-03-25 22:54:57 +00:00
Ralph Castain
522a23d6a3
A few changes to the FT-related configure options:
...
1. fix a bug that caused an infinite loop in configure when specifying want-ft but not want-ft-thread by removing a stale reference to the opal-progress-thread option
2. add want-ft=orcm so we can build the orcm errmgr component
3. cleanup the use of "ompi_want_ft_xxx" and replace it with "opal_want_ft_xxx" so that naming conventions are preserved
This commit was SVN r22885.
2010-03-25 22:53:48 +00:00
Josh Hursey
e4f2d03d28
ErrMgr Framework redesign to better support fault tolerance development activities.
...
Explained in more detail in the following RFC:
http://www.open-mpi.org/community/lists/devel/2010/03/7589.php
This commit was SVN r22872.
2010-03-23 21:28:02 +00:00
Ralph Castain
0b9552cd4e
Expand the ESS framework's API to include a new function "query_sys_info" that allows the caller to retrieve key-value pairs of info on the local system capabilities (e.g., cpu type/model). Have each daemon and the HNP "sense" that information and provide it to their local procs to avoid having every proc querying the system directly.
...
This commit was SVN r22870.
2010-03-23 20:47:41 +00:00
Josh Hursey
9e967a3a9b
Revert this change, since even in the CR case we want to reset this var to NULL.
...
Thanks to Jeff for the catch.
This commit was SVN r22868.
2010-03-23 19:55:21 +00:00
Shiqing Fan
9591680ec0
One of the binaries was generated from a wrong source.
...
This commit was SVN r22865.
2010-03-23 09:56:11 +00:00
Ralph Castain
62e751a95c
Add a tag
...
This commit was SVN r22862.
2010-03-22 15:46:00 +00:00
Ralph Castain
d49f93b743
Cleanup the initialization handshake for multicast apps
...
This commit was SVN r22855.
2010-03-19 20:15:01 +00:00
Ralph Castain
74bd4adc6b
Add some diagnostics, correctly check for existing channel
...
This commit was SVN r22854.
2010-03-19 08:21:01 +00:00
Ralph Castain
abbdc2b527
Pass the job family to tools that need to connect to specific HNPs
...
This commit was SVN r22853.
2010-03-19 04:01:33 +00:00
Ralph Castain
a479e6c320
Provide the sender's name for blocking recv's
...
This commit was SVN r22852.
2010-03-19 04:00:34 +00:00
Ralph Castain
8fb71c0fe6
Add some helpful defined values
...
This commit was SVN r22850.
2010-03-19 03:59:29 +00:00
Ralph Castain
e291fc2c69
With Jeff's help, get the libraries to link as required.
...
Update ompi_info and orte-info to include the new framework.
Fix some selection logic and a typo'd variable name
Still remains ompi_ignored until we complete testing
This commit was SVN r22848.
2010-03-18 02:12:59 +00:00
Ralph Castain
3cd96928a9
Use the OMPI_CHECK_PACKAGE macro to check both header file and library existence before building the component.
...
Still haven't gotten the right libraries linked in...so add ompi_ignore/unignore until we get it all fully integrated.
This commit was SVN r22843.
2010-03-17 00:46:12 +00:00
Ralph Castain
b400b84162
Merge in the modified thread configure option branch per today's telecon.
...
Remove the --enable-progress-threads option as this is no longer functional, and hardcode OPAL_ENABLE_PROGRESS_THREADS to 0.
Replace the --enable-mpi-threads option with --enable-mpi-thread-multiple as this is clearer as to meaning. This option automatically turns "on" opal thread support if it wasn't already so specified. If the user specifies --disable-opal-multi-threads --enable-mpi-thread-multiple, we will error out with a message
Add a new --enable-opal-multi-threads option that turns "on" opal thread support without doing anything wrt mpi-thread-multiple
This commit was SVN r22841.
2010-03-16 23:10:50 +00:00
Ralph Castain
ffd5be6aa1
Add a new framework to ORTE for saving and recovering state information. Two components are included that use the db or dbm library for storing the data, with a distributed hash table component coming later.
...
Note that each of these components will only be selected if specifically requested - otherwise, a "NULL" component will be used. The framework is only opened by the HNP and orteds, though neither is currently coded to save/restore state
This commit was SVN r22839.
2010-03-16 20:59:48 +00:00
Rainer Keller
814fb9399f
- Further patches for support on NetBSD (and DragonFly) by
...
Aleksej Saushev.
Dont use bash or bashism in shell scripts
We should use Posix' setpgid(0,0), which is equivalent to setpgrp().
This commit was SVN r22829.
2010-03-15 05:33:42 +00:00
Josh Hursey
e9b5162d79
Fix the configure logic for --with-ft so that it properly takes a comma separated list.
...
Many of the OPAL_ENABLE_FT should be OPAL_ENABLE_FT_CR, so fix those.
The OPAL Layer INC should call opal_output on restart so that it can refresh the string it prints to reflect the current pid/hostname which may have changed.
This commit was SVN r22824.
2010-03-12 23:57:50 +00:00
Josh Hursey
b43d621f30
Remove an errant '$' in the configure.m4 files. Was causing problems with configure.
...
This commit was SVN r22821.
2010-03-12 20:08:22 +00:00
Ralph Castain
c16cd10bb2
Save the username, if specified, for each node
...
This commit was SVN r22817.
2010-03-11 15:24:18 +00:00
Ralph Castain
7105207b1c
If we only have one app participating in a all_gather (which lies under a modex as well), then we need to ensure that the returned buffer has the proper packing order so it can be unpacked correctly.
...
This commit was SVN r22815.
2010-03-10 19:22:06 +00:00
Ralph Castain
7ebf72b4aa
Trivial cleanup
...
This commit was SVN r22813.
2010-03-10 18:24:38 +00:00
Ralph Castain
7fd7b7a8cc
Fix the load_balance mapper so that it sets the #procs in the job before attempting to compute vpids
...
This commit was SVN r22812.
2010-03-10 17:52:19 +00:00
Ralph Castain
17936e6e5f
Ensure we cleanly terminate if an executable cannot be found
...
This commit was SVN r22805.
2010-03-10 16:45:08 +00:00
Josh Hursey
b73237c92a
Identify the process sending the update in the verbose message (helps debugging of process control).
...
This commit was SVN r22804.
2010-03-10 00:23:24 +00:00
Shiqing Fan
49502af2ba
fix the type cast.
...
This commit was SVN r22800.
2010-03-09 10:02:50 +00:00
Ralph Castain
4355134991
Let the vm launcher specify the mapping policy
...
This commit was SVN r22797.
2010-03-08 19:13:21 +00:00
Ralph Castain
bfa39d7f7e
Update the seq mapper to support lists from -host. Reorg the dash_host code to provide an ordered list as required by the seq mapper
...
This commit was SVN r22795.
2010-03-08 09:54:49 +00:00
Ralph Castain
9e7f621a98
Port Brad's paffinity change to the 1.4 branch over to the trunk so we don't lose it going forward.
...
This commit was SVN r22794.
2010-03-07 18:44:22 +00:00
Ralph Castain
2a0f7e95ee
Don't double account for the killed local proc - only adjust num_local_procs when the proc actually dies.
...
This commit was SVN r22787.
2010-03-05 13:53:18 +00:00
Ralph Castain
b2e24693c4
Check the return status when we forward stdin and remove the recipient when they are no longer alive
...
This commit was SVN r22786.
2010-03-05 13:41:28 +00:00
Ralph Castain
577eef1491
Pretty-print the recvd command for debug purposes
...
This commit was SVN r22785.
2010-03-05 13:38:20 +00:00
Ralph Castain
cdae19cf7b
Add a convenience macro to make a job family
...
This commit was SVN r22784.
2010-03-05 13:35:09 +00:00
Ralph Castain
f2c65dc70f
Ensure that the errmgr does not take action if the process was terminated by a "kill_procs" command as this can lead to circular logic.
...
Cleanup the kill_procs command by removing a no-longer-used param. We update the process state when the proc actually exits.
This commit was SVN r22783.
2010-03-05 13:22:12 +00:00
Ralph Castain
ef6c432e22
Fix a nasty bug where we would hang if an application trapped signals such as SIGTERM - a permissible thing to do. In such cases, we removed the process from the waitpid system and then sent it a SIGTERM. If the application trapped that and attempted to cleanly terminate, it would send us a sync message - and the daemon would then add it back to its local child list, causing both the daemon and the process to hang.
...
In this revision, we let the process terminate/exit however it can, and then pick it up via the usual waitpid.
This commit was SVN r22781.
2010-03-05 04:14:56 +00:00
Shiqing Fan
db747e4390
Remove the old timing parameter but using orte_timing instead. Thanks for Rainer.
...
This commit was SVN r22775.
2010-03-04 15:00:03 +00:00
Ralph Castain
c88fe1ea54
Create a new mca parameter to control creation of session directories. Defaults to true so that the current behavior of always creating them is preserved. If set to false (0), then don't create session directories. Helps in those environments where session directories are a problem.
...
Tell the sm btl that it cannot run if no session directories were created.
This commit was SVN r22756.
2010-03-02 15:18:33 +00:00
Ralph Castain
cd1efbb41e
Try and do a better job of cleanup in abnormal termination. Ensure the daemons whack session directories prior to disabling signal traps. Ensure that the HNP and daemons all cleanup when they are doing an internal abort.
...
This commit was SVN r22755.
2010-03-02 14:51:23 +00:00
Ralph Castain
b692645772
Remote daemons should -always- whack any lingering session directories when exiting
...
This commit was SVN r22749.
2010-03-02 05:28:53 +00:00
Ralph Castain
69fe5ca69b
Correctly compute bynode mapping, even in the presence of a $#$%#@^$ rankfile
...
This commit was SVN r22748.
2010-03-02 05:21:42 +00:00
Ralph Castain
bef06d52bc
Silence compiler warning
...
This commit was SVN r22747.
2010-03-01 21:04:26 +00:00
Ralph Castain
5514d9c673
Fix the stupid rankfile mapper again, hopefully not breaking everything else to accommodate it. Looks like the round-robin mappers still work, at least...
...
This commit was SVN r22746.
2010-03-01 20:40:47 +00:00
Ralph Castain
96590b9fad
Filter multicast messages to avoid cross-job confusion
...
This commit was SVN r22729.
2010-02-28 18:22:56 +00:00
Ralph Castain
359dc5cad3
Complete the app_idx change by cleaning up warnings in mappers
...
This commit was SVN r22728.
2010-02-27 18:14:27 +00:00
Ralph Castain
2541aa98ab
Change the app_idx type to uint32_t to support users who use large numbers of app_contexts. Set it up as a new typedef so we can change it later without as much effort.
...
This commit was SVN r22727.
2010-02-27 17:37:34 +00:00
Ralph Castain
6c0d7940c7
Add a new MCA param (and corresponding mpirun cmd line option) to output the debugger proctable info after launch. The output is just the job map with the process pid included, so you get a node-by-node list of the process ranks on that node and thier pids.
...
Works for initial launch and comm_spawn. xml and non-xml output is available
This commit was SVN r22725.
2010-02-27 08:32:25 +00:00
Shiqing Fan
4a3f42d159
Correctly initialize the CCP command line buffer.
...
This commit was SVN r22721.
2010-02-26 15:53:00 +00:00
Ralph Castain
c6448587fe
It is okay to not select an rmcast module
...
This commit was SVN r22719.
2010-02-26 02:39:04 +00:00
Ralph Castain
b89a21f0fa
Grrr....cleanup the new module
...
This commit was SVN r22711.
2010-02-25 06:08:04 +00:00
Ralph Castain
8954700845
No, we don't have a .windows file...
...
This commit was SVN r22710.
2010-02-25 02:18:54 +00:00
Ralph Castain
18c7aaff08
Update the grpcomm framework to be more thread-friendly.
...
Modify the orte configure options to specify --enable-multicast such that it directs components to build or not instead of littering the code base with #if's. Remove those #if's where they used to occur.
Add a new grpcomm "mcast" module to support multicast operations. Still some work required to properly perform daemon collectives for comm_spawn operations. New module only builds when --enable-multicast is provided, and when specifically selected.
This commit was SVN r22709.
2010-02-25 01:11:29 +00:00
Jeff Squyres
af6f1f4b00
Add pkg-config(1) config files to Open MPI. Additionally, fix a minor
...
bug: libmpi_f90 had libmpi.la in its LIBADD instead of libmpi_f77.la.
Fixes trac:2244.
This commit was SVN r22704.
The following Trac tickets were found above:
Ticket 2244 --> https://svn.open-mpi.org/trac/ompi/ticket/2244
2010-02-24 18:46:06 +00:00
Shiqing Fan
44fe33452c
Use this option only on Windows, so protect it with #ifdef __WINDOWS__.
...
This commit was SVN r22701.
2010-02-24 08:50:03 +00:00
Jeff Squyres
d9b6b5af0c
This commit converts us to the "one big libmpi" scheme that has been
...
discussed extensively. See
https://svn.open-mpi.org/trac/ompi/ticket/2092 and the RFC thread
http://www.open-mpi.org/community/lists/devel/2010/02/7447.php .
Specifically:
* Create LT convenience libraries for OPAL and ORTE if the layer
above them is being created (use the already-defined
AM_CONDITIONALs to know if the project above us is being built).
* ORTE slurps in the LT convenience library for OPAL; OMPI slurps in
the LT convenience library for ORTE.
* Wrapper compilers now only -l one library (e.g., ortecc only does
-lopen-ret, and mpicc only does -lmpi).
This commit was SVN r22691.
2010-02-23 22:20:01 +00:00
Shiqing Fan
7a5a5ce024
Add a global option for inputing the head node name for Windows CCP trough command line.
...
This commit was SVN r22688.
2010-02-23 19:42:51 +00:00
Jeff Squyres
f65eebf53d
More changes for NetBSD. Thanks to Aleksej Saushev for this patch.
...
This commit was SVN r22680.
2010-02-22 15:05:09 +00:00
Ralph Castain
65a8ab4267
Cleanup the kill_procs command. Send a SIGTERM initially to allow C/R operations, and to be polite. Correctly update proc state if there is a problem so we don't hang.
...
The change to just using SIGKILL was originally done due to problems whereby waitpid thought a proc had died, but it hadn't. We'll continue debugging that problem separately, but SIGTERM is required for C/R to work properly.
This commit was SVN r22674.
2010-02-21 19:35:32 +00:00
Ralph Castain
2be03b4fb6
Cleanup a few bugs in the rmcast subsystem
...
This commit was SVN r22650.
2010-02-18 01:54:45 +00:00
Shiqing Fan
08ffdbe987
Changes for portable platform headers. Commit it on behalf of Ralph.
...
This commit was SVN r22619.
2010-02-15 22:14:59 +00:00
Ralph Castain
9a5fdbb622
Continue development of reliable multicast
...
This commit was SVN r22616.
2010-02-14 19:20:56 +00:00
Ralph Castain
58a1151566
Ensure the man page gets into the tarball
...
This commit was SVN r22613.
2010-02-13 02:39:10 +00:00
Ralph Castain
dc6de3e9b5
Add an "orte-info" tool to report ORTE and OPAL configuration and parameter info ala "ompi_info". Temporary solution until refactoring of the "info" system can be undertaken.
...
This commit was SVN r22608.
2010-02-11 23:02:14 +00:00
Josh Hursey
ec4498c258
update copyright
...
This commit was SVN r22605.
2010-02-11 14:03:36 +00:00
Josh Hursey
4513440df9
The ordering of the appfile matters. The ranks must be in increasing rank
...
order, so that they get assigned the same ORTE name on restart.
cmr:v1.4.2
cmr:v1.5
This commit was SVN r22586.
2010-02-09 00:31:24 +00:00
Eugene Loh
924bc10ceb
Add description of the sequential mapper (-mca rmaps seq) to the
...
mpirun man page.
This commit was SVN r22567.
2010-02-06 06:22:29 +00:00
Josh Hursey
a3583b8f57
Fix --bynode option to remember for subsequent jobs where it left off last time.
...
Add a ''map_bynode'' info key to determine if the job to be started by comm_spawn* should be mapped by node or by slot. Default is to map according to the default policy set when the parent job was started.
cmr:v1.5.1
This commit was SVN r22564.
2010-02-05 15:37:49 +00:00
Iain Bason
28f03a2d86
Suspend/resume enhancements:
...
Have orte call setpgrp after forking (but before exec) when
orte_forward_job_control is set. Then have it send signals to the
child's process group. This allows suspending jobs that fork.
If a SIGTSTP arrives before the processes have been launched, then
record it and suspend them right after launching.
This commit was SVN r22557.
2010-02-04 15:47:20 +00:00
Shiqing Fan
bbcf1f71c4
Remove a incorrect callback, which was based on the old source base without WMI. This makes no harm on 32 bit Windows, but it seems causing exceptions on 64 bit Windows sometimes. What this callback does is just waiting on the given pid which actually is a remote pid, so it won't work as expected.
...
cmr:v1.4.2
cmr:v1.5
This commit was SVN r22549.
2010-02-04 10:47:28 +00:00
Ralph Castain
c2aba2a6d7
Pack the resource key correctly
...
This commit was SVN r22541.
2010-02-03 18:21:27 +00:00
Ralph Castain
16b7bc7a82
Sigh...get the order right to match unpack
...
This commit was SVN r22539.
2010-02-03 15:50:43 +00:00
Ralph Castain
e88627a7ca
Ensure we don't go through rml open/select more than once.
...
Open the rml to get the uri when bootstrapping daemons
This commit was SVN r22538.
2010-02-03 15:38:32 +00:00
Ralph Castain
cb1007b5a9
Pass back the number of daemons in the system
...
This commit was SVN r22537.
2010-02-03 14:31:16 +00:00
Shiqing Fan
bdc13dacb1
A type cast.
...
This commit was SVN r22520.
2010-01-31 20:22:22 +00:00
Jeff Squyres
007a6c7b99
Per #2201 , move the user arguments up to be the first set of argv
...
after the compiler argv tokens.
Not closing #2201 yet; there's still discussion on that ticket about
whether we want to do more or not.
Refs trac:2201
cmr:v1.4.2
cmr:v1.5
This commit was SVN r22513.
The following Trac tickets were found above:
Ticket 2201 --> https://svn.open-mpi.org/trac/ompi/ticket/2201
2010-01-29 22:51:35 +00:00
Josh Hursey
4f56b4b133
revert mistaken commit r22511
...
This commit was SVN r22512.
The following SVN revision numbers were found above:
r22511 --> open-mpi/ompi@a16cb86181
2010-01-29 16:30:16 +00:00
Josh Hursey
a16cb86181
copyright fixes
...
This commit was SVN r22511.
2010-01-29 16:28:41 +00:00
Ralph Castain
7badff9d2d
Okay to return no available nodes for mapping when launching daemons - just means there is nothing to do
...
This commit was SVN r22509.
2010-01-28 22:58:28 +00:00
Ralph Castain
86dd1d41af
Handle zero-length iovecs in multicast messages
...
This commit was SVN r22507.
2010-01-28 15:29:43 +00:00
Ralph Castain
f66b6cae23
Enable the boot of an orted "virtual machine". Modify the mapper framework to allow mapping of only daemons. Remove the cm ras module as no longer required. Modify the orted code to always send back node arch info. Remove the "--enable-bootstrap" configure option as this feature will now always be available.
...
This commit was SVN r22480.
2010-01-25 22:25:13 +00:00
Josh Hursey
b749ecbab8
This commit fixes trac:2190.
...
Originally the patch was to improve the error message, but when digging into the code I found a subtle bug. If the daemon does not tell the HNP what CRS component it used, then the HNP tries to figure it out from the metadata (this is an uncommon case). The path the HNP used was not complete, so it was unable to find the metadata information. This patch fixes this by adding the 'snapshot_reference' to the 'snapshot_location' which completes the path for this search.
cmr:v1.4 (needs a custom patch)
cmr:v1.5
This commit was SVN r22479.
The following Trac tickets were found above:
Ticket 2190 --> https://svn.open-mpi.org/trac/ompi/ticket/2190
2010-01-25 20:28:38 +00:00
Josh Hursey
44497a0567
forgot to update the copyright refs trac:2194
...
This commit was SVN r22478.
The following Trac tickets were found above:
Ticket 2194 --> https://svn.open-mpi.org/trac/ompi/ticket/2194
2010-01-25 19:24:09 +00:00
Josh Hursey
36ab4a60b3
Improve the error message for when ompi-checkpoint cannot find a HNP to connect to.
...
This commit fixes trac:2189.
cmr:v1.4 (needs custom patch)
cmr:v1.5
This commit was SVN r22477.
The following Trac tickets were found above:
Ticket 2189 --> https://svn.open-mpi.org/trac/ompi/ticket/2189
2010-01-25 19:17:43 +00:00
Ralph Castain
e4bf33dcab
Just a slight efficiency improvement - why check a flag twice?
...
This commit was SVN r22472.
2010-01-23 03:57:56 +00:00
Shiqing Fan
c29a668e37
Remove flex.exe and its license file from the tarball.
...
cmr:v1.4
cmr:v1.5
This commit was SVN r22469.
2010-01-22 16:40:13 +00:00
Ralph Castain
00b493e10d
Make the sigpipe message be a verbose output
...
This commit was SVN r22464.
2010-01-22 03:46:11 +00:00
Ralph Castain
2517799102
Report, but ignore, SIGPIPE events. The odls already resets this signal handler when spawning local procs.
...
This commit was SVN r22463.
2010-01-21 05:01:06 +00:00
Shiqing Fan
4836e8878a
Update a few more CMake scripts.
...
This commit was SVN r22454.
2010-01-19 17:34:55 +00:00
Ralph Castain
3fe5e3e142
Propagate the user's callback data during non-blocking sends
...
This commit was SVN r22432.
2010-01-15 20:02:47 +00:00
Shiqing Fan
ad763c327d
Restore several linked libraries that were deleted by mistake in r22405.
...
This commit was SVN r22415.
The following SVN revision numbers were found above:
r22405 --> open-mpi/ompi@872a4047ba
2010-01-14 21:50:42 +00:00
Shiqing Fan
872a4047ba
Fix the bug that caused by ADD_DEPENDENCIES() from different version of CMake.
...
In CMake 2.6 and earlier, this function add dependencies for targets and also link the target libraries automatically, but in CMake 2.8,this behavior has been changed, i.e. it will only add the dependencies but no link, which will cause linking errors at compilation time.
This commit was SVN r22405.
2010-01-14 18:10:20 +00:00
Ralph Castain
cec840f6b9
The ability to add procs to a running job was unfortunately borked when we added the detection of a proc exiting before calling init. Re-enable it here, ensuring that procs that are being restarted and/or added to a job do -not- call barrier during orte_init.
...
This commit was SVN r22404.
2010-01-14 17:59:42 +00:00
Shiqing Fan
0259fa0b9c
Correct a few variable names.
...
This commit was SVN r22401.
2010-01-14 10:55:15 +00:00
Ralph Castain
adb2430e24
Missed one place, of course
...
This commit was SVN r22400.
2010-01-13 23:11:44 +00:00
Ralph Castain
c782c98433
Rename the "basic" rmcast component "udp" to more accurately reflect its operation
...
This commit was SVN r22399.
2010-01-13 23:01:25 +00:00
Ralph Castain
237eb4e8df
For some strange reason, every so often it appears possible for the event library to trip the read event on a socket, yet have the read itself yield an error. If/when that happens, report the error and continue on.
...
This happens rarely, but it does seem to happen.
This commit was SVN r22398.
2010-01-13 19:23:28 +00:00
Ralph Castain
ae1719306b
Fix a bug in non-blocking sends
...
This commit was SVN r22395.
2010-01-13 05:37:36 +00:00
Ralph Castain
b35486d945
The CM ess module needs to open the sysinfo framework and select modules prior to when others need it. Thus, setup a flag to avoid multiple open/select within that framework.
...
This commit was SVN r22393.
2010-01-12 22:03:49 +00:00
Ralph Castain
48486df4fe
Cleanup some diagnostics
...
This commit was SVN r22389.
2010-01-12 01:25:19 +00:00
Ralph Castain
9f3ccebeaa
We need to barrier for orte apps when the job is initially started, but we must not do the barrier when a proc is restarted as the other procs in the job won't know to participate.
...
This commit was SVN r22388.
2010-01-10 02:21:30 +00:00
Ralph Castain
16b16c5cb8
Fix a silly typo
...
This commit was SVN r22387.
2010-01-09 15:34:49 +00:00
Ralph Castain
e0afc30708
Since the decision on whether or not to use script wrapper compilers is a configure-time option, we need the wrapper compiler script to be in the tarball
...
This commit was SVN r22386.
2010-01-09 01:08:10 +00:00
Ralph Castain
add84178ef
Fix a silly typo that prevented tcp multicast messages from being delivered
...
This commit was SVN r22384.
2010-01-08 20:30:27 +00:00
Brian Barrett
86d8356b13
Updates to allow OMPI to build on Cray XT platforms running Catamount
...
This commit was SVN r22381.
2010-01-07 18:14:03 +00:00
Ralph Castain
09763ec711
Since we modified ORTE to declare that any process that terminates after calling "init" while at least one other process has not yet called "init" is an error, we have to ensure that non-MPI ORTE apps (i.e., apps that call orte_init but not mpi_init) include a barrier in orte_init. Otherwise, fast ORTE apps almost always wind up triggering the "abnormal termination" condition.
...
The barrier is protected with a test to ensure that MPI apps don't execute it and wind up doing two barriers during their init.
This commit was SVN r22378.
2010-01-07 06:58:01 +00:00
Ralph Castain
ef1bfaa823
Add the ability to track how many times a process has been restarted, and to communicate that value to a process when it is restarted in case it needs to take action when it is restarted as opposed to being started for the first time.
...
This commit was SVN r22377.
2010-01-07 01:19:44 +00:00
Ralph Castain
a12de9d1e8
Oh, the pain one little word can make...sigh.
...
This commit was SVN r22364.
2010-01-05 23:29:56 +00:00
Ralph Castain
5faf857840
Add a new tag for pnp/multicast send of direct messages
...
This commit was SVN r22352.
2009-12-31 20:34:58 +00:00
Ralph Castain
b3a58f8b83
Pass the correct address when packing iovec bytes for multicast.
...
Thanks to Rick Payne for the correction.
This commit was SVN r22351.
2009-12-30 20:59:31 +00:00
Ralph Castain
bb7aa9797f
Not sure why the nightly tarball had a problem as this wasn't changed, but comment out the orte wrapper compiler man pages for now
...
This commit was SVN r22350.
2009-12-30 02:52:29 +00:00
Ralph Castain
89a6131032
Check the return status code on all dss operations within the rmcast modules
...
This commit was SVN r22349.
2009-12-30 01:45:31 +00:00
Ralph Castain
fad1ba15b0
Move the test for case-sensitive file system from ompi to opal so that all layers can have that knowledge.
...
Use that for the orte wrapper compilers
This commit was SVN r22348.
2009-12-29 23:26:45 +00:00
Ralph Castain
d6003e3369
Update the mpirun man page to include info on the "!" option to the xterm cmd line option that holds the xterm window open after process termination so the user can see the process' output.
...
This commit was SVN r22341.
2009-12-26 16:01:03 +00:00
Ralph Castain
50074f0770
Remove unused (and uninitialized) variable
...
This commit was SVN r22340.
2009-12-24 01:36:47 +00:00
Ralph Castain
aaf1119f40
Garrr...ensure we accurately know when to update the contact info so we don't do it incorrectly as procs terminate, thus causing the system to think that perfectly good apps are incorrectly terminating.
...
Thanks to George for pointing out the problem
This commit was SVN r22332.
2009-12-17 20:40:21 +00:00
Ralph Castain
db2cbd3166
Okay, okay - do it at destruct time too.
...
This commit was SVN r22331.
2009-12-17 20:08:49 +00:00
Ralph Castain
a56e09c874
Per suggestion from Josh, init the sender field of the msg_packet object to INVALID
...
This commit was SVN r22330.
2009-12-17 20:03:35 +00:00
Ralph Castain
8ab962411c
Detect the scenario where one or more procs fail to call orte/ompi_init while others in the job do. This scenario can cause the job to hang as MPI_Init contains a barrier operation that will not complete. Although ORTE does not contain such a barrier, it still will be considered as an error scenario so that we can detect the MPI case - otherwise, ORTE has no knowledge of OMPI and wouldn't know how to differentiate the use-cases.
...
Take advantage of the changes to update the routed_base_receive code to avoid message overlap.
This commit was SVN r22329.
2009-12-17 19:39:53 +00:00
Ralph Castain
06d1f2cfe2
Add some new tests to the ORTE collection
...
This commit was SVN r22328.
2009-12-17 19:30:57 +00:00
Josh Hursey
313acba4ce
Move the mca_base_is_component_required() functionality to mca/base per suggestion so that it can be reused in other components.
...
This commit was SVN r22327.
2009-12-17 15:12:26 +00:00
Josh Hursey
a418a7dc43
Make sure to look in not only the env var, but also {{{orte_routed_base_components}}} to confirm that this is the only component available, and intended for selection.
...
This commit was SVN r22323.
2009-12-16 20:17:26 +00:00
Josh Hursey
646f90a90a
Small fix for a egde case
...
This commit was SVN r22322.
2009-12-16 18:06:05 +00:00
George Bosilca
a2310808f1
Santa's back! Fix all warnings about the deprecated usage of
...
stringWithCString as well as the casting issue between NSInteger and
%d. The first is solved by using stringWithUTF8String, which apparently
will always give the right answer (sic). The second is fixed as suggested
by Apple by casting the NSInteger (hint: which by definition is large
enough to hold a pointer) to a long and use %ld in the printf.
This commit was SVN r22317.
2009-12-16 00:06:37 +00:00
Ralph Castain
9acec283af
Add a new TCP module to the reliable multicast framework. This module uses ORTE's grpcomm.xcast functionality to "fake" multicasts for environments where regular multicast isn't reliable.
...
Modify the startup logic to allow for this use-case.
This commit was SVN r22310.
2009-12-15 01:18:27 +00:00
Ralph Castain
0ffa4f2f0c
Ensure we cancel the lingering recv in the allgather code to avoid having incorrect counters.
...
Thanks to Damien for spotting the problem.
This commit was SVN r22301.
2009-12-14 13:21:56 +00:00
Josh Hursey
4357159ac9
Make sure to check for the NO_CKPT state while waiting. This means that the target was not able to checkpoint [ever | at this time]. So {{{ompi-checkpoint}}} should exit after printing the error message, instead of hanging and waiting.
...
Will need to be moved to v1.5 and v1.4. v1.4 will require a custom patch, but should apply cleanly to v1.5. CMRs to follow.
This commit was SVN r22289.
2009-12-09 16:01:33 +00:00
George Bosilca
501d1cc4ad
Set default values to avoid using these variables uninitialized.
...
This commit was SVN r22279.
2009-12-08 18:42:22 +00:00
Ralph Castain
e3a2e66ec2
Add limits on rmcast seq numbers
...
This commit was SVN r22269.
2009-12-05 01:20:14 +00:00
Jeff Squyres
9fffd30660
Fix a typo in the man page. Thanks to Jeremiah Willcock for pointing
...
it out.
This commit was SVN r22268.
2009-12-04 21:10:50 +00:00