1
1
Граф коммитов

724 Коммитов

Автор SHA1 Сообщение Дата
Ralph Castain
98580c117b Introduce staged execution. If you don't have adequate resources to run everything without oversubscribing, don't want to oversubscribe, and aren't using MPI, then staged execution lets you (a) run as many procs as there are available resources, and (b) start additional procs as others complete and free up resources. Adds a new mapper as well as a new state machine.
Remove some stale configure.m4's we no longer need.

Optimize the nidmaps a bit by only sending info that has changed each time, instead of sending a complete copy of everything. Makes no difference for the typical MPI job - only impacts things like staged execution where we are sending multiple (possibly many) launch messages.

This commit was SVN r27165.
2012-08-28 21:20:17 +00:00
Ralph Castain
e0c39c94e8 Complete the cleanup of the preload files system. Remove the dest_dir option as moving things to arbitrary locations - especially absolute paths - can prove disastrous. Remove the preload_libs option as these can be treated as just files. Cleanup some of the pack/unpack code as the dss handles NULL strings just fine. Deal a little better with absolute paths, noting that tar now strips the leading '/' for us (showing my age as it didn't used to do so).
Remove the odls_base_state.c file as that code is now covered by the new broadcast form of preload_files.

This commit was SVN r27127.
2012-08-24 02:28:29 +00:00
Ralph Castain
b4a544ad2a Per discussion with Josh, use the --preload-xxx cmd line options to broadcast files to all nodes. Add --set-cwd-to-session-dir option to start procs in their session directories. Add OMPI_FILE_LOCATION envar to tell procs where their prepositioned files went.
This commit was SVN r27125.
2012-08-23 21:28:05 +00:00
Ralph Castain
a572b6fa9f Pick the right place
This commit was SVN r27085.
2012-08-17 00:28:28 +00:00
Ralph Castain
35fef87202 Make the "no virtual machine" selection more intuitive by providing a --novm option to mpirun.
This commit was SVN r27048.
2012-08-15 14:55:03 +00:00
Ralph Castain
589acf550c Improve the new MPI_INFO_ENV to better handle Java applications and to correctly report the info for singletons.
This commit was SVN r27025.
2012-08-13 22:13:49 +00:00
Jeff Squyres
3719b6c68b After some further discussion between Jeff, Ralph, and Josh, rever
r26951.  The feeling is that fixing the actual problem of the command
line parser not always identifying when invalid command line options
were specified (i.e., r26953) was a better solution.

This commit was SVN r26979.

The following SVN revision numbers were found above:
  r26951 --> open-mpi/ompi@1f8df92c3c
  r26953 --> open-mpi/ompi@0b7b3feba9
2012-08-09 20:56:01 +00:00
Ralph Castain
1f8df92c3c Remove the confusion over which options are "to" and which are "by" by creating synonyms so that either spelling works.
This commit was SVN r26951.
2012-08-05 14:40:38 +00:00
Ralph Castain
c7f9a0fa34 Check for recursive use of mpirun - issue error message and abort if detected
This commit was SVN r26903.
2012-07-28 21:50:56 +00:00
Abhishek Kulkarni
5c58a1c9c1 Fix C/R support in the trunk.
Among other things, this patch deals with the following issues:
* fix ompi-checkpoint argument parsing
* ompi-restart -showme prints an extraneous "Restarted child with PID" 
  message. Move around the debug statement to avoid this.
* fixes for the state machine changes

This commit was SVN r26770.
2012-07-09 23:34:13 +00:00
Ralph Castain
e6f3586415 Remove the orte notifier framework, per discussion at the devel meeting and follow-up with Jeff (who took the action item)
This commit was SVN r26637.
2012-06-22 18:09:23 +00:00
Brian Barrett
9af72072a3 Use MKDIR_P instead of mkdir_p in Makefiles, as MKDIR_P is the only one
defined in recent versions of AC/AM.

This commit was SVN r26625.
2012-06-21 16:52:37 +00:00
Ralph Castain
0a713cd27e Add database framework to ORTE and refactor modex code to utilize it. Create the "hash" db component from the prior modex db code. Leave the other components ignored for now - will activate them later.
Modex is still a blocking operation at this point.

This commit was SVN r26618.
2012-06-19 13:38:42 +00:00
Ralph Castain
269cb2b8d9 Some cleanup to remove calls to opal_progress when running with orte progress threads, and to ensure that all orte-related events are in the orte event base.
This commit was SVN r26591.
2012-06-11 19:59:53 +00:00
Ralph Castain
d6279fc971 Fix the debugger daemon launch support to fit the new state machine. Treat debugger daemons just like any other job, except that we map them only to nodes where an app process currently exists (as opposed to every node in the system). Trigger breakpoint and rank0 release only after the debugger daemons are in position.
This commit was SVN r26556.
2012-06-06 02:01:23 +00:00
Jeff Squyres
99c5afb397 Remove clang compiler warnings.
This commit was SVN r26523.
2012-05-29 23:36:06 +00:00
Ralph Castain
be6ed9c2df Allow partial use of allocations by specifying the max number of daemons (i.e., max VM size) for the job
This commit was SVN r26499.
2012-05-27 16:48:19 +00:00
Jeff Squyres
7969faf372 Fixes trac:3057: minor update to the man page to state that slot locations
in rankfiles use ''physical'' device indexes (vs. logical indexes).

This commit was SVN r26478.

The following Trac tickets were found above:
  Ticket 3057 --> https://svn.open-mpi.org/trac/ompi/ticket/3057
2012-05-23 11:43:33 +00:00
Ralph Castain
b217124bd8 Symlink instead of copy
This commit was SVN r26464.
2012-05-21 23:07:48 +00:00
Ralph Castain
da3873af6f Rename the mapreduce tool to "mr+" per the marketing types
This commit was SVN r26463.
2012-05-21 21:17:44 +00:00
Ralph Castain
a526afae92 Ensure we always cleanup local procs, no matter how we exited.
This commit was SVN r26454.
2012-05-18 23:37:40 +00:00
Ralph Castain
12ebc0e269 Don't need this to be a bin program as the class is captured in the jar
This commit was SVN r26453.
2012-05-18 23:37:18 +00:00
Ralph Castain
b16e43f489 Silence a warning on Mac
This commit was SVN r26449.
2012-05-18 15:27:04 +00:00
Ralph Castain
ca1b325738 Tweak the java setup so it works better on Mac. Only build mapreduce and allocators if hadoop support was requested.
This commit was SVN r26448.
2012-05-18 01:02:01 +00:00
Jeff Squyres
2d78728d38 Fix the macro name in the comment: it's EXTRA_DIST, not EXTRA_SOURCES.
This commit was SVN r26429.
2012-05-10 14:07:36 +00:00
Jeff Squyres
b325c17c72 It's a little weird to put in a blank _SOURCES line for the
HDFSFileFinder PROGRAM, but if we don't put in a _SOURCES line at all,
Automake will default to "HDFSFileFinter_class_SOURCES =
HDFSFileFinder.c", which clearly will cause problems.  

But we don't want to put the .java file in _SOURCES, either, because
we haven't configured Automake to handle Java (because current
versions of Automake only have GCJ, not other Java compilers).  So set
HDFSFileFinder_class_SOURCES to blank and list the .java file in
EXTRA_SOURCES (so that they get picked up for "make dist").

This commit was SVN r26424.
2012-05-10 13:54:51 +00:00
Ralph Castain
640f0610aa Fix the makefile to install the perl scripts properly
This commit was SVN r26416.
2012-05-09 14:06:02 +00:00
Ralph Castain
fd796cce0a Add an allocator tool for finding HDFS file locations and obtaining allocations for those nodes (supports both Hadoop 1 and 2). Split the Java support into two parts: detection of Java support and request for Java bindings.
This commit was SVN r26414.
2012-05-09 01:13:49 +00:00
Jeff Squyres
2ba10c37fe Per RFC, bring in the following changes:
* Remove paffinity, maffinity, and carto frameworks -- they've been
   wholly replaced by hwloc.
 * Move ompi_mpi_init() affinity-setting/checking code down to ORTE.
 * Update sm, smcuda, wv, and openib components to no longer use carto.
   Instead, use hwloc data.  There are still optimizations possible in
   the sm/smcuda BTLs (i.e., making multiple mpools).  Also, the old
   carto-based code found out how many NUMA nodes were ''available''
   -- not how many were used ''in this job''.  The new hwloc-using
   code computes the same value -- it was not updated to calculate how
   many NUMA nodes are used ''by this job.''
   * Note that I cannot compile the smcuda and wv BTLs -- I ''think''
     they're right, but they need to be verified by their owners.
 * The openib component now does a bunch of stuff to figure out where
   "near" OpenFabrics devices are.  '''THIS IS A CHANGE IN DEFAULT
   BEHAVIOR!!''' and still needs to be verified by OpenFabrics vendors
   (I do not have a NUMA machine with an OpenFabrics device that is a
   non-uniform distance from multiple different NUMA nodes).
 * Completely rewrite the OMPI_Affinity_str() routine from the
   "affinity" mpiext extension.  This extension now understands
   hyperthreads; the output format of it has changed a bit to reflect
   this new information.
 * Bunches of minor changes around the code base to update names/types
   from maffinity/paffinity-based names to hwloc-based names.
 * Add some helper functions into the hwloc base, mainly having to do
   with the fact that we have the hwloc data reporting ''all''
   topology information, but sometimes you really only want the
   (online | available) data.

This commit was SVN r26391.
2012-05-07 14:52:54 +00:00
Ralph Castain
b2f77bf08f Extend the iof by adding two new components to support map-reduce IO chaining. Add a mapreduce tool for running such applications.
Fix the state machine to support multiple jobs being simultaneously launched as this is not only required for mapreduce, but can happen under comm-spawn applications as well.

This commit was SVN r26380.
2012-05-02 21:00:22 +00:00
Ralph Castain
a927318ea1 Add -N option as synonym for "npernode"
This commit was SVN r26367.
2012-05-01 16:18:14 +00:00
Jeff Squyres
501a86afe1 No need to include the generated files in the tarball. Thanks to
Eugene for pointing this out.

This commit was SVN r26339.
2012-04-25 14:19:18 +00:00
Ralph Castain
5d14fa7546 Fix mpi_abort, minimize error output.
This commit was SVN r26266.
2012-04-11 14:37:08 +00:00
Ralph Castain
14d5525fb1 Some minor cleanups. Get singletons working. Cleanup abort handling so it gets properly identified.
This commit was SVN r26261.
2012-04-10 19:08:54 +00:00
Ralph Castain
bd8b4f7f1e Sorry for mid-day commit, but I had promised on the call to do this upon my return.
Roll in the ORTE state machine. Remove last traces of opal_sos. Remove UTK epoch code.

Please see the various emails about the state machine change for details. I'll send something out later with more info on the new arch.

This commit was SVN r26242.
2012-04-06 14:23:13 +00:00
Ralph Castain
6dc44dc4b8 Look at the basename of the appname for the "java" keyword
This commit was SVN r26190.
2012-03-24 00:38:18 +00:00
Josh Hursey
a595525366 Add the callers name to the 'comm failed' error message, so we know between which two peers the communication failed.
This commit was SVN r26117.
2012-03-08 21:55:19 +00:00
Ralph Castain
c3cf46af65 Ensure install_dirs are filled in before parsing prefix
This commit was SVN r26093.
2012-03-03 23:14:15 +00:00
Ralph Castain
b8f093d1a0 Switch precedence - take the --prefix value over the absolute-path-to-mpirun so the backend prefix can be different from that of mpirun on hetero machines.
This commit was SVN r26085.
2012-03-02 22:59:13 +00:00
Ralph Castain
6c93dd13b0 Cleanup the prefix handling by mpirun. Important note: we do NOT support per-app_context prefixes!!
Don't let app_files trump given prefix values. Assign according to following precedence rules:

1. absolute path to mpirun, if given
2. --prefix value, if given to mpirun
3. default prefix, if configured with --enable-orterun-prefix-default
4. prefix from first app in app_file, if given
5. no prefix

This commit was SVN r26081.
2012-03-02 19:48:25 +00:00
Jeff Squyres
97b3603036 A bunch of fixes and improvements to Open MPI's various command line tools.
* fixed some bugs where "unknown" tokens were allowed on the command
   line (which should really only be used for ortertun).
 * if an unknown token is encountered, print a short error to stderr
   and quit with a nonzero exit status
 * if we don't find the right number of parameters to an option, print
   a short error to stderr and quit with a nonzero exit status
 * when --help is given, print the help message to stdout (not stderr)
   and quit with a zero exit status
 * added --showme:help option to the wrapper compilers
 * updated docs in opal/util/cmd_line.h
 * other small/miscellaneous CLI parsing bugs in various tools

I won't bore you with what we did before.  :-)  Here's some examples
of what the new behavior looks like:

{{{
% ompi_info --bogus
ompi_info: Error: unknown option "--bogus"
Type 'ompi_info --help' for usage.
% ompi_info --param bogus
ompi_info: Error: option "--param" did not have enough parameters (2)
Type 'ompi_info --help' for usage.
%
}}}

This commit was SVN r26072.
2012-02-29 17:52:38 +00:00
Ralph Castain
bc5886707f Document the mpirun exit status behavior
This commit was SVN r26009.
2012-02-22 23:47:00 +00:00
Ralph Castain
47c64ec837 Roll in Java bindings per telecon discussion. Man pages still under revision
This commit was SVN r25973.
2012-02-20 22:12:43 +00:00
Ralph Castain
d7d8a8cdf7 Some cleanup of the tmpdir session directory specifications. Remove the --tmpdir option from orterun as it was confusing. Create an orte_local_tmpdir_base mca param in its place. Clarify the role of the local vs remote vs global tmpdir base params, and ensure that you don't set conflicting options.
Remove the OMPI_PREFIX_ENV environmental variable as that was totally confusing as a way of setting a tmpdir base location.

This commit was SVN r25941.
2012-02-16 16:10:01 +00:00
Jeff Squyres
54cf60eb4b $(RM) is not a standard macro. Just use "rm" -- every platform has it.
This commit was SVN r25934.
2012-02-15 19:51:59 +00:00
Jeff Squyres
ae9503db6e Remove the sentence that says that --prefix is a per-context option.
This commit was SVN r25932.
2012-02-15 18:31:27 +00:00
Ralph Castain
61ac2bb11b If no session directories are being created, then we cannot create the debugger attachment fifo - so don't complain about it.
This commit was SVN r25802.
2012-01-27 04:05:23 +00:00
Ralph Castain
6db8c56cd4 Add local and node ranks to debugger daemon procs so the odls properly launches them
This commit was SVN r25774.
2012-01-25 03:17:10 +00:00
Ralph Castain
bf09133631 Correctly track the number of debugger daemons being spawned
This commit was SVN r25741.
2012-01-19 18:17:07 +00:00
Ralph Castain
6235a355de Correctly handle co-spawning of daemons when attaching to a running job. We cannot use the general process mappers as we only want debugger daemons spawned on nodes where application procs already exist. So custom build the map for the debugger daemon job, and have the plm just launch that job without doing its usual vm-spawn step.
This commit was SVN r25736.
2012-01-18 00:19:49 +00:00