1
1
Граф коммитов

10485 Коммитов

Автор SHA1 Сообщение Дата
Jeff Squyres
fff1057597 We do ''not'' want the orted to yield the processor more than
necessary (it already uses blocking semantics while waiting for events
on fd's, so it's not taking more cycles from MPI applications than
necessary), or this can/will cause lengthy delays in orted processing.
The problem we most recently saw was with routed OOB messages getting
significantly delayed before being delivered to the target MPI
processes.  This was problematic for BTLs that use OOB messaging for
wireup.  There is a lengthy comment in orted_main.c that describes
this in more detail.

Setting the orted to not voluntarily call yield() prevents this bad
behavior.  The orted only runs in small bursts anyway (and blocks the
rest of the time), so this is not harmful to application performance.

This commit was SVN r16365.
2007-10-06 09:24:51 +00:00
Ralph Castain
54b2cf747e These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.

This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:

As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.

In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.

The incoming changes revamp these procedures in three ways:

1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.

The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.

Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.


2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.

The size of this data has been reduced in three ways:

(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.

To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.

(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.

(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.

While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.


3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.

It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.

Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.


There are a few minor additional changes in the commit that I'll just note in passing:

* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.

* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.

* cleanup of some stale header files

This commit was SVN r16364.
2007-10-05 19:48:23 +00:00
Jelena Pjesivac-Grbovic
ada43fef9e This fixes bug #1157 in coll/self module.
All vector functions had incorrect handling of the offset.

This commit was SVN r16360.
2007-10-05 17:40:16 +00:00
Jeff Squyres
f92154fc72 Gah -- ompi_info doesn't setup the connect pseudo component, so it'll
be NULL.  Ensure to protect for this.

This commit was SVN r16333.
2007-10-04 18:03:56 +00:00
Jeff Squyres
13fa7ae93e It's not necessary to link against all 3 libs (in fact, we shouldn't
do it -- let libtool pull them in via the .la file if it needs to)

This commit was SVN r16332.
2007-10-04 18:01:30 +00:00
Jeff Squyres
80ce974291 Fixes trac:1156: ensure to finalize the "connect" sub-component.
This commit was SVN r16330.

The following Trac tickets were found above:
  Ticket 1156 --> https://svn.open-mpi.org/trac/ompi/ticket/1156
2007-10-04 17:36:12 +00:00
Andrew Friedley
2e66590993 Fix mistakes in the basic component.. can't call collectives on the communicator and always pass the basic module.. have to give them the module off the communicator.
This commit was SVN r16329.
2007-10-04 16:29:24 +00:00
Galen Shipman
77f080575f fix for the cray..
This commit was SVN r16317.
2007-10-03 19:25:23 +00:00
Jeff Squyres
7b0fe8b152 Revert r15900; the variable was already named correctly. This fixes
static builds for OMPI components that required extra LIBS or LDFLAGS
(e.g., the openib BTL).

Fixes trac:1155.

This commit was SVN r16314.

The following SVN revision numbers were found above:
  r15900 --> open-mpi/ompi@50941ec389

The following Trac tickets were found above:
  Ticket 1155 --> https://svn.open-mpi.org/trac/ompi/ticket/1155
2007-10-03 06:46:39 +00:00
Andrew Friedley
5be7f5e2dc fixes trac:1154
Check if an exclusion string (i.e. '-mca btl ^sm) was provided; if so OFUD just disables itself.

This commit was SVN r16307.

The following Trac tickets were found above:
  Ticket 1154 --> https://svn.open-mpi.org/trac/ompi/ticket/1154
2007-10-02 20:37:16 +00:00
Tim Mattox
ed7fd5ad90 Put in a 1.2.5 NEWS section on the trunk.
This commit was SVN r16301.
2007-10-02 14:33:46 +00:00
Tim Prins
34966edaf1 remove unneeded and never-initialized lock. The orte_ns.assign_tag function does all the locking we need for us.
This commit was SVN r16299.
2007-10-02 14:22:29 +00:00
Gleb Natapov
60af46d541 We have QP description in component structure, module structure and endpoint.
Each one of them has a field to store QP type, but this is redundant.
Store qp type only in one structure (the component one).

This commit was SVN r16272.
2007-09-30 16:14:17 +00:00
Gleb Natapov
9c04b127f5 Forget to put this fix in previous commit.
This commit was SVN r16271.
2007-09-30 15:33:20 +00:00
Gleb Natapov
3a15d645be Remove lcl_qp_attr from endpoint qp description. It is used during init only.
This commit was SVN r16270.
2007-09-30 15:29:35 +00:00
Brian Barrett
3a0067249c The previous hack to deal with Libtool not speaking Objective C stopped
working with Automake 1.10.  This is a new hack, which should be much
more flexible.  The ras doesn't contain any Objective C, so remove the
hack entirely from that Makefile.am.

This commit was SVN r16269.
2007-09-30 03:40:25 +00:00
Rolf vandeVaart
a87267ef92 Fix a build error on Solaris. MAXHOSTNAMELEN is defined in netdb.h.
This commit was SVN r16268.
2007-09-28 20:15:28 +00:00
Brian Barrett
48c49cb89c Handle case where modex_recv_string() isn't implemented (ie, the Cray)
This commit was SVN r16267.
2007-09-28 18:50:37 +00:00
Tim Prins
1d1d0f6d4c Fix segfault when user provides a working directory for comm_spawn. Thanks to Murat Knecht for reporting this and suggesting a fix.
This commit was SVN r16266.
2007-09-27 23:30:40 +00:00
Tim Prins
b161732af9 Be sure to restore the library flags in case of error. Thanks to Ake Sandgren for pointing this out.
This commit was SVN r16263.
2007-09-27 21:35:52 +00:00
Josh Hursey
665a1e280b Copyright updates that should have gone into r16252.
(Someday I'll learn to do this before committing)

This commit was SVN r16260.

The following SVN revision numbers were found above:
  r16252 --> open-mpi/ompi@e10f476c87
2007-09-27 14:37:04 +00:00
Josh Hursey
e10f476c87 Bring over the jjh-filem branch which contains a non-blocking FileM interface
and implementation. This has shown drastic performance benefit when
transferring Many files at roughly the same time.

I tested this for many different filem operations and everything was working
fine. Let me know if you have any problems with this functionality.

Some Notes:
 - opal-checkpoint now has a 'quiet' flag to keep it from being too verbose.

 - FileM RSH component is fully non-blocking.

 - FileM RSH component has incomming connection throttling since by default
   ssh only allows 10 concurrent scp connections to any single host. This
   default can be adjusted via an MCA parameter.
    {{{-mca filem_rsh_max_incomming 10}}}

 - There is an MCA parameter for max outgoing connections, but it is currently
   not implemented. If someone needs it then it should not be hard to implement.
    {{{-mca filem_rsh_max_outgoing 10}}}

 - Changed the FileM request structure so that it is a bit more explicit and
   flexible.

 - Moved the 'preload-binary' and 'preload-files' functionality into odls/base
   allowing for code reuse in the 'process' and 'default' ODLS components.

 - Fixed a bug in the process name resolution which broke the 'preload-*'
   functionality due to GPR table structure changes.

 - The FileM RSH component might be able to see even more speedup from using a
   thread pool to operate on the work_pool structures, but that is for future
   work.

 - Added a 'opal-show-help' file to ODLS Base

This commit was SVN r16252.
2007-09-27 13:13:29 +00:00
Aurelien Bouteiller
670956e172 Another cast mistake.
This commit was SVN r16247.
2007-09-26 21:14:35 +00:00
Aurelien Bouteiller
f7d7d58fb6 Various cast type errors on 64bit architectures
This commit was SVN r16246.
2007-09-26 20:54:18 +00:00
Brian Barrett
56e26ed390 Need to install the mpool_rdma.h so that we can build external BTLs that
use the RDMA protocol

This commit was SVN r16237.
2007-09-26 16:58:54 +00:00
Andrew Friedley
069e6dc4a0 Fix a bug introduced when the collective selection logic was changed to allow for a different component to be used for each collective.
Passing the barrier module to the bcast function is a bad idea when barrier is using a different component from bcast..

This commit was SVN r16212.
2007-09-25 17:09:52 +00:00
Pak Lui
97e692d85a mqs_communicator type should not be changed as it serves as the interface
between Totalview and DLL.

This commit was SVN r16200.
2007-09-24 19:02:56 +00:00
Josh Hursey
b5fc722c35 Add a flag to 'pretend' to do filem in snapc. This is useful when doing
performance characterization, and should not be used by anyone doing anything
else since it will not produce a globally consistent checkpoint in this mode.

This commit was SVN r16192.
2007-09-24 16:19:45 +00:00
Gleb Natapov
c7105eadc7 Update Voltaire copyright.
This commit was SVN r16189.
2007-09-24 10:11:52 +00:00
Tim Mattox
b45a5e2fdc Elaborated on Message Queue entry in the Open MPI 1.2.4 NEWS.
This commit was SVN r16184.
2007-09-21 20:33:40 +00:00
Aurelien Bouteiller
0df0087f17 Investigating improvement of cache line management on shared memory
This commit was SVN r16183.
2007-09-21 20:02:56 +00:00
Tim Mattox
909f10b8c0 Add Message Queue entry to the Open MPI 1.2.4 NEWS.
This commit was SVN r16182.
2007-09-21 19:47:23 +00:00
Josh Hursey
1fe1276fd5 Make sure to match on the communicator ID as well.
This commit was SVN r16179.
2007-09-21 18:16:02 +00:00
Josh Hursey
3e51d7bb25 Implement the MPI_Iprobe and MPI_Probe wrappers.
Remove some old, unused code.

This commit was SVN r16178.
2007-09-21 16:28:46 +00:00
George Bosilca
8bdd14ba40 Remove unique_id which wasn't used anymore. Instead use the recv_context which
is set to the cid of the communicator (unique id for each communicator).
Make sure each communicator have a group attached to it. The MPI_COMM_NULL
should have the MPI_GROUP_NULL as a group, in all circumstances.

This commit was SVN r16177.
2007-09-21 14:30:40 +00:00
Aurelien Bouteiller
d3b376a340 This patch adds actual non-blocking sender-based message logging. This improves bandwidth. Still need to work on malloc/mmap storage to reach optimal bandwidth.
This commit was SVN r16172.
2007-09-21 03:24:08 +00:00
Aurelien Bouteiller
bc318b35e2 There is room in convertor to copy the packed data. It works just need to add the correct memcopy. It does not manage the short messages but I alreqdy think of a workaround for this (and it might even be better regarding latency).
This commit was SVN r16169.
2007-09-20 21:57:21 +00:00
Pak Lui
54c87daaed Fix a SEGV when the user updates the message queue graph after the user
executable has called MPI_Finalize(). It happens when removing the group
from each of the communicators, that MPI_COMM_NULL doesn't have a group. 
Also fix the code from skipping over every other communicator when
freeing the groups.

This commit was SVN r16166.
2007-09-20 18:58:16 +00:00
Tim Prins
38fde640ad Fix builds on FreeBSD by renaming strings.h to f77_strings.h so that our file does not get accidently included by FreeBSD's string.h.
Thanks to Karol Mroz for pointing out the problem.

This commit was SVN r16164.
2007-09-19 23:24:23 +00:00
Tim Prins
e25bb7f187 Some platforms (such as FreeBSD) need libutil.h included for openpty.
Thanks to Karol Mroz for pointing this out.

This commit was SVN r16163.
2007-09-19 21:59:22 +00:00
Jeff Squyres
0e131a54bf Updates to the README.
This commit was SVN r16162.
2007-09-19 17:48:15 +00:00
Jeff Squyres
1d7910f398 Fix a few typos, clarify a few bullets.
This commit was SVN r16160.
2007-09-19 13:41:56 +00:00
Aurelien Bouteiller
bbac6e650a New improved version of sender-based. Under dev but a new framework for expressing various methods have been added.
This commit was SVN r16159.
2007-09-19 03:42:56 +00:00
Brian Barrett
6bf121e17b fix comment
This commit was SVN r16154.
2007-09-18 16:30:45 +00:00
Gleb Natapov
097b17d30e Prevent a receive request from been freed while other thread holds a reference
to it or there is an outstanding completion for the request.

This commit was SVN r16153.
2007-09-18 16:18:47 +00:00
Tim Mattox
164a577908 Add another entry to the Open MPI 1.2.4 NEWS.
This commit was SVN r16150.
2007-09-18 15:47:46 +00:00
Aurelien Bouteiller
f762850242 Split run_global into process_project and process_framework. This allows for calling only process framework to create components internal sub-frameworks
Minor change to ompi_mca.m4 to move AC_CONFIG_FILES(framework/makefile) in autogen process (instead of configure process), where we still now the actual framework path (instead of guessing using $project/mca/$framework). 

This have shown no side effects in our testing. Let us know if this breaks one of your components in some exotic context. 

This commit was SVN r16146.
2007-09-18 10:36:08 +00:00
Jeff Squyres
f9b9beba77 Allow the LSF components to be shipped in the nightly tarball and open
it up to others.

This commit was SVN r16143.
2007-09-17 22:42:33 +00:00
Jeff Squyres
33955a0ed0 Oops -- when converted from uint to int, -1 (the default value,
meaning "infinite") is no longer larger than the minimum required
size.  So put in an appropriate test to ensure that "infinite" was not
requested. 

This commit was SVN r16142.
2007-09-17 19:28:21 +00:00
Jeff Squyres
130a272cec Fix some compiler warnings about signed/unsigned comparisons.
This commit was SVN r16139.
2007-09-17 13:08:45 +00:00