1
1
Граф коммитов

1451 Коммитов

Автор SHA1 Сообщение Дата
Adrian Knoth
42d5fe62f9 Fixed misplaced #endif
This commit was SVN r17028.
2008-01-01 11:02:38 +00:00
Jeff Squyres
213b5d5c6e Per long threads on the mailing list and much confusion discussion
about linkers, have all OPAL, ORTE, and OMPI components '''not'' link
against the OPAL, ORTE, or OMPI libraries.

See ttp://www.open-mpi.org/community/lists/users/2007/10/4220.php for
details (or https://svn.open-mpi.org/trac/ompi/wiki/Linkers for a
better-formatted version of the same info).

This commit was SVN r16968.
2007-12-15 13:32:02 +00:00
Josh Hursey
f7812baf5b forgot a bit of error checking in the last commit
This commit was SVN r16953.
2007-12-13 14:41:18 +00:00
Josh Hursey
a287c9cb65 This commit distinguishes the file transfer stage from the finish stage.
This commit also cleans up the checkpoint and terminate case making it more
precise than before. Previously the application could make a small amount of
progress between checkpoint completion and application termination. Now the
application will make no progress at all in this time span.

Additional minor change:
 - Start using OPAL_INT_TO_BOOL instead of if/else logic

This commit was SVN r16952.
2007-12-13 14:37:17 +00:00
Rolf vandeVaart
3ea89b69ae Remove a few tabs. Allow the output stream to be
passed to the close command for verbose output.  This
matches all the other frameworks.

This commit was SVN r16938.
2007-12-11 20:44:56 +00:00
Jeff Squyres
d47ea89181 George rightly pointed out that this should be 0600, not 0660.
This commit was SVN r16927.
2007-12-11 12:55:08 +00:00
Jeff Squyres
1640897272 Ensure to use the 3rd argument to open(), per suggestion from
Sebastian Schmitzdorff, because Fedora 8 no longer accepts the
2-argument form.

This commit was SVN r16923.
2007-12-10 22:19:23 +00:00
Josh Hursey
27c9016b93 sleep -> usleep so we can be a bit more eager when waiting for events to finish.
Still working on solutions that do not involve sleeping, but this will do for
now.

This commit was SVN r16824.
2007-12-03 19:27:32 +00:00
Jeff Squyres
c20350b943 Patch submitted by Brian Barrett, inspired by this thread:
http://www.open-mpi.org/community/lists/users/2007/11/4547.php.

- Better handling of ECONNABORTED from connect on Linux.
- Reduce extraneous output from OOB when TCP connections must
  be retried.

This commit was SVN r16808.
2007-11-30 21:42:15 +00:00
Ron Brightwell
edb9d8e354 Added Catamount to the conditional compilation since Catamount
doesn't support fork() or pipe() either.  This removes a
linker warning message when building for Cray XT with Catamount.

This commit was SVN r16772.
2007-11-21 21:37:58 +00:00
Ethan Mallove
005652c9d4 * Embed ident strings into the Open MPI libraries using one of the following
methods (in order of precedence):
  1. #pragma ident <ident string> (e.g., Intel and Sun)
  1. #ident <ident string> (e.g., GCC)
  1. static const char ident[] = <ident string> (all others)
By default, the ident string used is the standard Open MPI version string. Only
the following libraries will get the embedded version strings (e.g., DSOs will
not):
  * libmpi.so
  * libmpi_cxx.so
  * libmpi_f77.so
  * libopen-pal.so
  * libopen-rte.so
* Added two new configure options:
  * `--with-package-name="STRING"` (defaults to "Open MPI username@hostname
    Distribution"). `STRING` is displayed by `ompi_info` next to the "Package"
    heading.
  * `--with-ident-string="STRING"` (defaults to the standard Open MPI version
    string - e.g., X.Y.Zr######). `%VERSION%` will expand to the Open MPI
    version string if it is supplied to this configure option.

This commit was SVN r16644.
2007-11-03 02:40:22 +00:00
George Bosilca
d67c0eefb4 Remove a compilation warning about using uninitialized variables.
This commit was SVN r16589.
2007-10-26 20:15:28 +00:00
George Bosilca
b1b5cb6453 Looks like SO_REUSEPORT it's not defined on some platforms. Switch
to the conventional SO_REUSEADDR instead.

This commit was SVN r16588.
2007-10-26 19:56:21 +00:00
George Bosilca
337f78a4a8 Restrict the port range for the OOB and the BTL. Each protocols (v4 and v6)
has his own range which is defined by a min value and a range. By default
there is no limitation on the port range, which is exactly the same
behavior as before.

This commit was SVN r16584.
2007-10-26 16:36:51 +00:00
Jeff Squyres
9e4387d021 * Use new BEGIN_C_DECLS / END_C_DECLS convention
* Add newline at end of file to avoid compiler warning

This commit was SVN r16579.
2007-10-26 13:40:38 +00:00
Shiqing Fan
3c38c9c020 - Add extern "C" to resolve linkage specification problems.
This commit was SVN r16577.
2007-10-26 09:54:42 +00:00
Ralph Castain
a791ce2299 The processor affinity must be set on a per-process basis, not per-app-context.
This commit was SVN r16559.
2007-10-23 20:46:16 +00:00
George Bosilca
7a63f9b730 I somehow mess up my last commit. Sorry.
This commit was SVN r16543.
2007-10-22 15:08:17 +00:00
George Bosilca
b93f72bdfd Remove 2 warnings about uninitialized i and quit_flags.
This commit was SVN r16542.
2007-10-22 15:01:15 +00:00
Jeff Squyres
5637c7a5a0 In addition to r16513, this commit fixes trac:1170.
If we cannot resolve the route to the peer that we're trying to send
to, don't queue up the message in the TCP OOB -- instead, return it to
the upper layer (e.g., the RML) and let it decide what to do.

In the case of the routed RML, the tree component will queue it up for
later transmission.  Hence, we don't want the message queued up both
here in the TCP OOB and the tree routed.  Also see some more
discussion / explanation in #1171.

This commit was SVN r16540.

The following SVN revision numbers were found above:
  r16513 --> open-mpi/ompi@7ae9589d70

The following Trac tickets were found above:
  Ticket 1170 --> https://svn.open-mpi.org/trac/ompi/ticket/1170
2007-10-22 13:46:57 +00:00
Jeff Squyres
7ae9589d70 The header is at the address of the buffer pointed to by the iov, not
the address of the iov.

This commit was SVN r16513.
2007-10-19 12:40:14 +00:00
Jeff Squyres
abf1b728b9 Minor code maintenance fix -- put the THREAD_UNLOCK outside the if
statement so that you only have to have it once.

This commit was SVN r16512.
2007-10-19 12:36:26 +00:00
Ralph Castain
73eeb7f0d2 Fix a bug in the way we handled buffer releases and the conditioned wait that held us in the xcast until completed.
This commit was SVN r16504.
2007-10-19 01:17:01 +00:00
Josh Hursey
0bf61a1b84 Move in some accumulated small features and minor bug fixes for C/R support.
{{{
svn merge -r 16447:16475 https://svn.open-mpi.org/svn/ompi/tmp/jjh-fgs .
}}}

This commit was SVN r16478.
2007-10-17 13:47:36 +00:00
Ralph Castain
6f498c0964 Since we no longer set the APP_NUM attribute in the case of a singleton, protect the code so we don't crash in that case.
This commit was SVN r16461.
2007-10-16 16:17:48 +00:00
Ralph Castain
ec5fe78876 When in the unity message routing mode, we have to update the RML contact info in the parent procs so that they know how to talk to the children. Ideally, this would be done in the MPI layer since that layer knows which procs are actively involved in the comm_spawn. However, it isn't being done there, which causes comm_spawn to fail, so do it explicitly in the RTE.
Note that this means ALL procs in the parent job are updated, even though they may not be participating in the comm_spawn. This doesn't really hurt anything - just unnecessary.

Comm_spawn still has a problem when a child process shares a node with a parent, so this doesn't fix everything. It only fixes the bug of ensuring all procs know how to talk to each other.

This commit was SVN r16460.
2007-10-16 16:09:41 +00:00
Ralph Castain
713b6e13a5 Improve diagnostic output messages when errors are hit
This commit was SVN r16457.
2007-10-16 14:51:52 +00:00
Josh Hursey
ea0652d20f If we are going to pretend to do filem, then we should always pretend.
No one should be using this feature except for me. :)

This commit was SVN r16454.
2007-10-15 20:04:35 +00:00
Ralph Castain
b6196e8a39 When we can detect that a daemon has failed, then we would like to terminate the system without having it lock up. The "hang" is currently caused by the system attempting to send messages to the daemons (specifically, ordering them to kill their local procs and then terminate). Unfortunately, without some idea of which daemon has died, the system hangs while attempting to send a message to someone who is no longer alive.
This commit introduces the necessary logic to avoid that conflict. If a PLS component can identify that a daemon has failed, then we will set a flag indicating that fact. The xcast system will subsequently check that flag and, if it is set, will send all messages direct to the recipient. In the case of "kill local procs" and "terminate", the messages will go directly to each orted, thus bypassing any orted that has failed.

In addition, the xcast system will -not- wait for the messages to complete, but will return immediately (i.e., operate in non-blocking mode). Orterun will wait (via an event timer) for a period of time based on the number of daemons in the system to allow the messages to attempt to be delivered - at the end of that time, orterun will simply exit, alerting the user to the problem and -strongly- recommending they run orte-clean.

I could only test this on slurm for the case where all daemons unexpectedly died - srun apparently only executes its waitpid callback when all launched functions terminate. I have asked that Jeff integrate this capability into the OOB as he is working on it so that we execute it whenever a socket to an orted is unexpectedly closed. Meantime, the functionality will rarely get called, but at least the logic is available for anyone whose environment can support it.

This commit was SVN r16451.
2007-10-15 18:00:30 +00:00
Jeff Squyres
423f23eb6a Fixes trac:1160. There is still some other problem in the OOB, but we
wanted to commit this to get wider testing.

This commit was SVN r16445.

The following Trac tickets were found above:
  Ticket 1160 --> https://svn.open-mpi.org/trac/ompi/ticket/1160
2007-10-15 15:41:36 +00:00
Josh Hursey
f16a42947a Change some default MCA parameters:
- Global snapshot directory = $HOME
 - FileM 'rsh' = 'ssh'
 - FileM 'rcp' = 'scp'

This commit was SVN r16444.
2007-10-15 15:21:17 +00:00
Josh Hursey
520c27ac94 If the HNP is acting as the orted for local launch then the gpr_replica
variable is not defined. Make sure to set it to something reasonable 
so that file preloading still works (instead of seg faulting :)

Thanks to Hiep Bui Hoang for reporting this bug.

This commit was SVN r16433.
2007-10-11 19:47:04 +00:00
Josh Hursey
e483c36cea Remove a big of debug in filem/rsh that should have never been committed.
A guesture towards overlapping file removal with metadata update.

This commit was SVN r16432.
2007-10-11 19:37:33 +00:00
Josh Hursey
31e9369e8b Fix orterun so it does not get influenced by an application's argv set.
For example, if I have an application that, internal to the application, takes
the argument '-mca foo bar' we do not want orterun to pick up this argument and
pass it through the system.

So the following
{{{
 shell$ mpirun -np 2 -mca btl tcp,self ./myapp -mca foo bar
}}}

orterun should pick up {{{-mca btl tcp,self}}} but not {{{-mca foo bar}}} which
it was previous to this commit.

I tested command line runs and runs with app files to confirm this patch works.

This commit was SVN r16431.
2007-10-11 18:33:40 +00:00
Ralph Castain
3dbd4d9be7 Squeeeeeeze the launch message. This is the message sent to the daemons that provides all the data required for launching their local procs. In reorganizing the ODLS framework, I discovered that we were sending a significant amount of unnecessary and repeated data. This commit resolves this by:
1. taking advantage of the fact that we no longer create the launch  message via a GPR trigger. In earlier times, we had the GPR create the launch message based on a subscription. In that mode of operation, we could not guarantee the order in which the data was stored in the message - hence, we had no choice but to parse the message in a loop that checked each value against a list of possible "keys" until the corresponding value was found.

Now, however, we construct the message "by hand", so we know precisely what data is in each location in the message. Thus, we no longer need to send the character string "keys" for each data value any more. This represents a rather large savings in the message size - to give you an example, we typically would use a 30-char "key" for a 2-byte data value. As you can see, the overhead can become very large.

2. sending node-specific data only once. Again, because we used to construct the message via subscriptions that were done on a per-proc basis, the data for each node (e.g., the daemon's name, whether or not the node was oversubscribed) would be included in the data for each proc. Thus, the node-specific data was repeated for every proc.

Now that we construct the message "by hand", there is no reason to do this any more. Instead, we can insert the data for a specific node only once, and then provide the per-proc data for that node. We therefore not only save all that extra data in the message, but we also only need to parse the per-node data once.

The savings become significant at scale. Here is a comparison between the revised trunk and the trunk prior to this commit (all data was taken on odin, using openib, 64 nodes, unity message routing, tested with application consisting of mpi_init/mpi_barrier/mpi_finalize, all execution times given in seconds, all launch message sizes in bytes):

Per-node scaling, taken at 1ppn:

#nodes           original trunk                         revised trunk
             time               size                time               size
      1      0.10                819                0.09                564
      2      0.14               1070                0.14                677
      3      0.15               1321                0.14                790
      4      0.15               1572                0.15                903
      8      0.17               2576                0.20               1355
     16      0.25               4584                0.21               2259
     32      0.28               8600                0.27               4067
     64      0.50              16632                0.39               7683

Per-proc scaling, taken at 64 nodes

   ppn             original trunk                         revised trunk
              time               size                time               size
      1       0.50              16669                0.40               7720
      2       0.55              32733                0.54              11048
      3       0.87              48797                0.81              14376
      4       1.0               64861                0.85              17704


Condensing those numbers, it appears we gained:

per-node message size: 251 bytes/node -> 113 bytes/node

per-proc message size: 251 bytes/proc  -> 52 bytes/proc

per-job message size:  568 bytes/job -> 399 bytes/job 
(job-specific data such as jobid, override oversubscribe flag, total #procs in job, total slots allocated)

The fact that the two pre-commit trunk numbers are the same confirms the fact that each proc was containing the node data as well. It isn't quite the 10x message reduction I had hoped to get, but it is significant and gives much better scaling.

Note that the timing info was, as usual, pretty chaotic - the numbers cited here were typical across several runs taken after the initial one to avoid NFS file positioning influences.

Also note that this commit removes the orte_process_info.vpid_start field and the handful of places that passed that useless value. By definition, all jobs start at vpid=0, so all we were doing is passing "0" around. In fact, many places simply hardwired it to "0" anyway rather than deal with it.

This commit was SVN r16428.
2007-10-11 15:57:26 +00:00
Rolf vandeVaart
25c95c9ee9 Fix build on solaris. Need to include sys/wait.h.
This commit was SVN r16426.
2007-10-11 15:04:30 +00:00
Jeff Squyres
e2df42eea3 Move the <sys/wait.h> below "orte_config.h"
This commit was SVN r16424.
2007-10-11 11:31:09 +00:00
George Bosilca
7cc9f588a8 Decorate the base functions with ORTE_DECLSPEC.
This commit was SVN r16423.
2007-10-11 00:02:49 +00:00
Ralph Castain
53af94fd87 Modify the configure system so that gridengine support is only built in specific conditions:
1. --with-sge, always builds
2. --without-sge, never builds
3. if neither is specified, build if and only if either SGE_ROOT is set or "qrsh" is found in the path

This commit was SVN r16422.
2007-10-10 21:39:16 +00:00
Josh Hursey
6e5341c659 Forgot to move a header in the code movement.
This commit was SVN r16420.
2007-10-10 15:39:40 +00:00
Ralph Castain
82a8e2d10d Reorganize the odls framework to place common functionality in the base, thus making maintenance easier. We still need this to be a framework as some environments (e.g., bproc) require significantly different functionality. However, there is quite a bit of commonality across the components, so this ensures that fixes in one get propagated across the others.
This patch also fixes a minor bug discovered along the way: we had "lost" the passing of the oversubscribed condition flag from the mapper to the orteds. Thus, we were not setting sched_yield correctly when in oversubscribed conditions (except when a hostfile was specified - different logic there because we treat the number of slots allocated on the node as "uncertain")

I did not modify the process component in this patch - I will send a proposed patch to the maintainers of that component so they can review it first.

This commit was SVN r16418.
2007-10-10 15:02:10 +00:00
Josh Hursey
7f833a9cb2 silence a warning that is triggered on restart
This commit was SVN r16417.
2007-10-10 14:25:49 +00:00
Ethan Mallove
d0b61db65c Add in a missing #include for Solaris builds.
This commit was SVN r16416.
2007-10-10 12:49:15 +00:00
Josh Hursey
aa8391f888 Local and global coordinators should be the only ones involved in the
movement of checkpoint files. This reduces the overhead on the applicaiton.

This commit was SVN r16412.
2007-10-09 19:52:47 +00:00
Galen Shipman
fda1306807 revert my stupidity..
This commit was SVN r16410.
2007-10-09 19:01:20 +00:00
Josh Hursey
8fe2ef5647 a missing include
This commit was SVN r16402.
2007-10-09 14:32:36 +00:00
Josh Hursey
7437f37e96 This commit contains the following:
* Fix some missing includes in a few places.
 * Add the cr_request() functionality to the BLCR CRS component.
   We are now dependent upon the 0.6.* series of BLCR.
 * Made the CR notification mechanism a registered function.
   This way we can have an OPAL-only version and it can be replaced at
   runtime with the ORTE version.
 * Add a 'opal_cr_allow_opal_only' parameter that will enable OPAL-only
   CR functionality when the user wants it. Default: Disabled.
 * Fix the placement of a checkpoint request check in MPI_Init
 * Pull the OPAL notification mechanism into the SnapC framework.
   * We no longer fork/exec the 'opal-checkpoint' command for local
   checkpointing, the Local coordinator in the orted does this directly.
   * The Local and Application coordinator talk together bypassing the OPAL
   notifiation mechanism.
   * Optimized the Local <-> App Coordinator communication.
   * Improved the structure used to track vpid_snapshots in the local coord.
 * Fix a race condition in which an application under heavy communication load
   may produce an inconsistent global checkpoint.

This commit was SVN r16389.
2007-10-08 20:53:02 +00:00
Galen Shipman
1c1b9d5480 make cray happy
This commit was SVN r16377.
2007-10-08 14:31:59 +00:00
Jeff Squyres
fff1057597 We do ''not'' want the orted to yield the processor more than
necessary (it already uses blocking semantics while waiting for events
on fd's, so it's not taking more cycles from MPI applications than
necessary), or this can/will cause lengthy delays in orted processing.
The problem we most recently saw was with routed OOB messages getting
significantly delayed before being delivered to the target MPI
processes.  This was problematic for BTLs that use OOB messaging for
wireup.  There is a lengthy comment in orted_main.c that describes
this in more detail.

Setting the orted to not voluntarily call yield() prevents this bad
behavior.  The orted only runs in small bursts anyway (and blocks the
rest of the time), so this is not harmful to application performance.

This commit was SVN r16365.
2007-10-06 09:24:51 +00:00
Ralph Castain
54b2cf747e These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.

This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:

As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.

In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.

The incoming changes revamp these procedures in three ways:

1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.

The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.

Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.


2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.

The size of this data has been reduced in three ways:

(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.

To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.

(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.

(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.

While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.


3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.

It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.

Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.


There are a few minor additional changes in the commit that I'll just note in passing:

* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.

* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.

* cleanup of some stale header files

This commit was SVN r16364.
2007-10-05 19:48:23 +00:00