1
1
Граф коммитов

1073 Коммитов

Автор SHA1 Сообщение Дата
Galen Shipman
1c1b9d5480 make cray happy
This commit was SVN r16377.
2007-10-08 14:31:59 +00:00
Ralph Castain
54b2cf747e These changes were mostly captured in a prior RFC (except for #2 below) and are aimed specifically at improving startup performance and setting up the remaining modifications described in that RFC.
The commit has been tested for C/R and Cray operations, and on Odin (SLURM, rsh) and RoadRunner (TM). I tried to update all environments, but obviously could not test them. I know that Windows needs some work, and have highlighted what is know to be needed in the odls process component.

This represents a lot of work by Brian, Tim P, Josh, and myself, with much advice from Jeff and others. For posterity, I have appended a copy of the email describing the work that was done:

As we have repeatedly noted, the modex operation in MPI_Init is the single greatest consumer of time during startup. To-date, we have executed that operation as an ORTE stage gate that held the process until a startup message containing all required modex (and OOB contact info - see #3 below) info could be sent to it. Each process would send its data to the HNP's registry, which assembled and sent the message when all processes had reported in.

In addition, ORTE had taken responsibility for monitoring process status as it progressed through a series of "stage gates". The process reported its status at each gate, and ORTE would then send a "release" message once all procs had reported in.

The incoming changes revamp these procedures in three ways:

1. eliminating the ORTE stage gate system and cleanly delineating responsibility between the OMPI and ORTE layers for MPI init/finalize. The modex stage gate (STG1) has been replaced by a collective operation in the modex itself that performs an allgather on the required modex info. The allgather is implemented using the orte_grpcomm framework since the BTL's are not active at that point. At the moment, the grpcomm framework only has a "basic" component analogous to OMPI's "basic" coll framework - I would recommend that the MPI team create additional, more advanced components to improve performance of this step.

The other stage gates have been replaced by orte_grpcomm barrier functions. We tried to use MPI barriers instead (since the BTL's are active at that point), but - as we discussed on the telecon - these are not currently true barriers so the job would hang when we fell through while messages were still in process. Note that the grpcomm barrier doesn't actually resolve that problem, but Brian has pointed out that we are unlikely to ever see it violated. Again, you might want to spend a little time on an advanced barrier algorithm as the one in "basic" is very simplistic.

Summarizing this change: ORTE no longer tracks process state nor has direct responsibility for synchronizing jobs. This is now done via collective operations within the MPI layer, albeit using ORTE collective communication services. I -strongly- urge the MPI team to implement advanced collective algorithms to improve the performance of this critical procedure.


2. reducing the volume of data exchanged during modex. Data in the modex consisted of the process name, the name of the node where that process is located (expressed as a string), plus a string representation of all contact info. The nodename was required in order for the modex to determine if the process was local or not - in addition, some people like to have it to print pretty error messages when a connection failed.

The size of this data has been reduced in three ways:

(a) reducing the size of the process name itself. The process name consisted of two 32-bit fields for the jobid and vpid. This is far larger than any current system, or system likely to exist in the near future, can support. Accordingly, the default size of these fields has been reduced to 16-bits, which means you can have 32k procs in each of 32k jobs. Since the daemons must have a vpid, and we require one daemon/node, this also restricts the default configuration to 32k nodes.

To support any future "mega-clusters", a configuration option --enable-jumbo-apps has been added. This option increases the jobid and vpid field sizes to 32-bits. Someday, if necessary, someone can add yet another option to increase them to 64-bits, I suppose.

(b) replacing the string nodename with an integer nodeid. Since we have one daemon/node, the nodeid corresponds to the local daemon's vpid. This replaces an often lengthy string with only 2 (or at most 4) bytes, a substantial reduction.

(c) when the mca param requesting that nodenames be sent to support pretty error messages, a second mca param is now used to request FQDN - otherwise, the domain name is stripped (by default) from the message to save space. If someone wants to combine those into a single param somehow (perhaps with an argument?), they are welcome to do so - I didn't want to alter what people are already using.

While these may seem like small savings, they actually amount to a significant impact when aggregated across the entire modex operation. Since every proc must receive the modex data regardless of the collective used to send it, just reducing the size of the process name removes nearly 400MBytes of communication from a 32k proc job (admittedly, much of this comm may occur in parallel). So it does add up pretty quickly.


3. routing RML messages to reduce connections. The default messaging system remains point-to-point - i.e., each proc opens a socket to every proc it communicates with and sends its messages directly. A new option uses the orteds as routers - i.e., each proc only opens a single socket to its local orted. All messages are sent from the proc to the orted, which forwards the message to the orted on the node where the intended recipient proc is located - that orted then forwards the message to its local proc (the recipient). This greatly reduces the connection storm we have encountered during startup.

It also has the benefit of removing the sharing of every proc's OOB contact with every other proc. The orted routing tables are populated during launch since every orted gets a map of where every proc is being placed. Each proc, therefore, only needs to know the contact info for its local daemon, which is passed in via the environment when the proc is fork/exec'd by the daemon. This alone removes ~50 bytes/process of communication that was in the current STG1 startup message - so for our 32k proc job, this saves us roughly 32k*50 = 1.6MBytes sent to 32k procs = 51GBytes of messaging.

Note that you can use the new routing method by specifying -mca routed tree - if you so desire. This mode will become the default at some point in the future.


There are a few minor additional changes in the commit that I'll just note in passing:

* propagation of command line mca params to the orteds - fixes ticket #1073. See note there for details.

* requiring of "finalize" prior to "exit" for MPI procs - fixes ticket #1144. See note there for details.

* cleanup of some stale header files

This commit was SVN r16364.
2007-10-05 19:48:23 +00:00
Brian Barrett
3a0067249c The previous hack to deal with Libtool not speaking Objective C stopped
working with Automake 1.10.  This is a new hack, which should be much
more flexible.  The ras doesn't contain any Objective C, so remove the
hack entirely from that Makefile.am.

This commit was SVN r16269.
2007-09-30 03:40:25 +00:00
Rolf vandeVaart
a87267ef92 Fix a build error on Solaris. MAXHOSTNAMELEN is defined in netdb.h.
This commit was SVN r16268.
2007-09-28 20:15:28 +00:00
Josh Hursey
665a1e280b Copyright updates that should have gone into r16252.
(Someday I'll learn to do this before committing)

This commit was SVN r16260.

The following SVN revision numbers were found above:
  r16252 --> open-mpi/ompi@e10f476c87
2007-09-27 14:37:04 +00:00
Josh Hursey
e10f476c87 Bring over the jjh-filem branch which contains a non-blocking FileM interface
and implementation. This has shown drastic performance benefit when
transferring Many files at roughly the same time.

I tested this for many different filem operations and everything was working
fine. Let me know if you have any problems with this functionality.

Some Notes:
 - opal-checkpoint now has a 'quiet' flag to keep it from being too verbose.

 - FileM RSH component is fully non-blocking.

 - FileM RSH component has incomming connection throttling since by default
   ssh only allows 10 concurrent scp connections to any single host. This
   default can be adjusted via an MCA parameter.
    {{{-mca filem_rsh_max_incomming 10}}}

 - There is an MCA parameter for max outgoing connections, but it is currently
   not implemented. If someone needs it then it should not be hard to implement.
    {{{-mca filem_rsh_max_outgoing 10}}}

 - Changed the FileM request structure so that it is a bit more explicit and
   flexible.

 - Moved the 'preload-binary' and 'preload-files' functionality into odls/base
   allowing for code reuse in the 'process' and 'default' ODLS components.

 - Fixed a bug in the process name resolution which broke the 'preload-*'
   functionality due to GPR table structure changes.

 - The FileM RSH component might be able to see even more speedup from using a
   thread pool to operate on the work_pool structures, but that is for future
   work.

 - Added a 'opal-show-help' file to ODLS Base

This commit was SVN r16252.
2007-09-27 13:13:29 +00:00
Josh Hursey
b5fc722c35 Add a flag to 'pretend' to do filem in snapc. This is useful when doing
performance characterization, and should not be used by anyone doing anything
else since it will not produce a globally consistent checkpoint in this mode.

This commit was SVN r16192.
2007-09-24 16:19:45 +00:00
Jeff Squyres
f9b9beba77 Allow the LSF components to be shipped in the nightly tarball and open
it up to others.

This commit was SVN r16143.
2007-09-17 22:42:33 +00:00
Shiqing Fan
d4a7fb1378 - A small fix of format.
This commit was SVN r16138.
2007-09-17 12:10:04 +00:00
George Bosilca
d32a54d74e There is no values[1] ... How did the compilers goes away with this !!!
This commit was SVN r16132.
2007-09-14 21:33:25 +00:00
George Bosilca
6897926dce Not used anymore.
This commit was SVN r16129.
2007-09-14 21:20:19 +00:00
Ralph Castain
45986ad2aa Add support to signal application procs for LSF
This commit was SVN r16120.
2007-09-13 18:09:14 +00:00
Ralph Castain
9fa254c017 Provide a better error message when a daemon unexpectedly dies under SLURM so we differentiate between fail to start and aborting while the app is running.
This commit was SVN r16115.
2007-09-12 20:53:50 +00:00
Josh Hursey
b4c68c0925 Turn back on the absolute path protection for the moment.
It is masking a bug that I'm tracking down in the SNAPC FULL - FILEM interations

Also make sure to cleanout the filem structure before asking for another
checkpoint file when not storing the files in place.

This commit was SVN r16109.
2007-09-12 18:19:39 +00:00
George Bosilca
e5d316dba6 Coverty: fix issues with using a string once it get freed. The problem, is that the
mca_base_register_string don't set the result to NULL is an error occurs.

This commit was SVN r16108.
2007-09-12 18:16:53 +00:00
Ralph Castain
f80ea093a2 Ensure that the orteds do not directly respond to USR1/2 signals. Those signals are trapped by mpirun and propagated from there - at most, the orteds are involved in the propagation process, but should never do anything on their own.
This commit was SVN r16098.
2007-09-12 14:32:31 +00:00
Shiqing Fan
548a4fe943 - Use IOVBASE_TYPE instead of char to avoid warnings on some systems.
This commit was SVN r16092.
2007-09-11 16:24:23 +00:00
Shiqing Fan
c1065d8262 - Some more type casts.
This commit was SVN r16087.
2007-09-11 11:28:43 +00:00
Brian Barrett
cfe737d1f9 Fix some mistaken error checks -- errors will be less than zero, not
greater than zero

This commit was SVN r16008.
2007-08-29 18:52:51 +00:00
Jeff Squyres
5628084fec Fix Coverity CID 463: remove unused variable / dead code.
This commit was SVN r15999.
2007-08-29 01:30:15 +00:00
Brian Barrett
dcf678dbab Fix heterogeneous issue with non-blocking RML receive, where the sender
field could be in the wrong endianness

This commit was SVN r15989.
2007-08-28 20:54:52 +00:00
Josh Hursey
729c63cf9d Fix invalid MCA 'base' names so they appear in ompi_info.
A subset of this patch needs to be applied to v1.2

Refs trac:928

This commit was SVN r15918.

The following Trac tickets were found above:
  Ticket 928 --> https://svn.open-mpi.org/trac/ompi/ticket/928
2007-08-18 03:05:45 +00:00
Brian Barrett
8294f6de03 The portals_utcp component doesn't actually need the POrtals libraries
and only pokes at environment variables.  So don't link in the libraries,
as that causes a whole other set of problems.

This commit was SVN r15899.
2007-08-17 03:48:39 +00:00
Andrew Friedley
2eedcd2539 Fixes trac:1047
Tie stdin to /dev/null to prevent stdin from being closed and thus making stdin not work in slurm allocations.

This commit was SVN r15892.

The following Trac tickets were found above:
  Ticket 1047 --> https://svn.open-mpi.org/trac/ompi/ticket/1047
2007-08-16 20:49:27 +00:00
Tim Prins
5a795128af Change it so that different components in orte use unique rml tags
This commit was SVN r15881.
2007-08-16 14:02:35 +00:00
Brian Barrett
330003361b * Free memory from asprintf
* need to compare ERANGE to errno

This commit was SVN r15860.
2007-08-14 21:12:00 +00:00
Brian Barrett
881dd0654e * Provide a hook so that a PLS can tell the orted it's starting that it
needs to override the default umask.  By default, this is not used
    since most environments do what the user would expect without any
    help.
  * Have TM use the newly added umask hook, so that processes inherit
    the user's umask from mpirun rather than the pbs_mom's umask, which
    the user has no control over.

This commit was SVN r15858.
2007-08-14 18:44:52 +00:00
Shiqing Fan
eea712f9ab - Export those components in correct way.
This commit was SVN r15804.
2007-08-08 16:20:17 +00:00
Brian Barrett
59524a9009 Fix issue where we set state to SHUTDOWN rather than CONNECTING when we
had to switch socket types.

This commit was SVN r15784.
2007-08-06 22:55:41 +00:00
Ralph Castain
eb3a97f428 Don't overwrite the local rank key
This commit was SVN r15776.
2007-08-06 16:56:23 +00:00
Shiqing Fan
d10570786c - A small fix, add missed flag parameters.
This commit was SVN r15774.
2007-08-06 16:15:38 +00:00
Josh Hursey
755658694e Bring in changes to support Cray's Compute Node Linux (CNL) and
Application Level Placement Scheduler (ALPS).

This commit was tested under two Cray machines at ORNL: Jaguar (Catamount)
and Rizzo (CNL Test cage). Both machines performed as they should across
the commit.

It is likely that mor changes will follow this the work and environment
stabilizes.

Most of the infrastructure works the same for Catamount and CNL
except for a few bits. Below are the highlights:

Default IFACE Change:
 On Catamount we can use PTL_IFACE_DEFAULT, but on the CNL system we have access
 to will fail on this interface, and should be set to:
    IFACE_FROM_BRIDGE_AND_NALID(PTL_BRIDGE_UK,PTL_IFACE_SS).
 So if we detect that we are running with YOD then use the former interface
 and if we detect that we are running with ALPS then use the latter.
 We will want to pursue a more elegant solution if this interface continues to 
 change across machines.

PtlGetId and cnos_register_ptlid:
 The header suggests that these should never be called when launching with YOD.
 But in the ALPS environment the cnos_barrier() will hang forever if these 
 functions are not called after PtlNIInit(). Since these functions only need to
 be called once, and the orte rmgr/cnos component is loaded before the ompi 
 common/portals componet then just call these functions once in the rmgr/cnos
 component.

cnos_barrier_init():
 This is a noop for YOD, but critical for ALPS. So be sure to call it before
 calling the first barrier in the rmgr/cnos component.

cnos_barrier vs cnos_pm_barrier:
 It is suggested the cnos_pm_barrier only be used during finalization 
 as it will indicate to the launcher (yod or aprun) that the app is about
 to complete. It was suggested that we use the regular cnos_barrier() instead.
 I want to look into this a bit more to make sure there are not adverse
 side effects. A note has been placed in the code to indicate this reasoning.

This commit was SVN r15756.
2007-08-03 19:46:38 +00:00
Jeff Squyres
106beff744 Ahem. Apparently we should be checking for ORTE_EQUAL upon return
from orte_ns.compare_fields(), not 0 (yes, they're the same [today],
but it is much better to check for symbolic names...).

This commit was SVN r15731.
2007-08-01 18:59:37 +00:00
Jeff Squyres
8d4b6c7b0d The HNP changing into an orted brought a bug in the iof svc component
to light: we weren't ack'ing properly for streams that originated (or
originated via proxy) and terminated within the HNP.  This commit
fixes that.

It also fixes a few style issues, and added some more opal_outputs for
debugging.  Also, fixed a bug where the fact that we forwarded (and
therefore might need to update the ack) was not correctly reported if
there were multiple forwards (which there are not as the system is
currently using IOF, but there could be).

Refs trac:1098 -- want to get another pair of eyes to look at this before
I close the ticket.

This commit was SVN r15730.

The following Trac tickets were found above:
  Ticket 1098 --> https://svn.open-mpi.org/trac/ompi/ticket/1098
2007-08-01 18:38:03 +00:00
Ralph Castain
066ff38d42 Ensure we read all the reported URI contact info when we fork an HNP for singleton support
This commit was SVN r15714.
2007-07-31 18:55:08 +00:00
George Bosilca
2e2bf472ff Mark the orte_abort function as noreturn and change the return value from
int to void. This function call exit at the end, so there is no way to
return from there. Apply the same thing to the errmsg_abort function and
update all components.

This commit was SVN r15704.
2007-07-31 16:09:52 +00:00
Sven Stork
855434de59 - fixes several coverty issues
- add missing initialisation for variables
  - use strncpy instead of strcpy

This commit was SVN r15683.
2007-07-30 14:44:37 +00:00
Rainer Keller
2c5d07217d - Coverity: use snprintf, instead of sprintf....
This commit was SVN r15669.
2007-07-29 11:23:23 +00:00
Jeff Squyres
3858cf48c0 Stop using the deprecated ORTE_NAME_ARGS() and switch to
ORTE_NAME_PRINT().

This commit was SVN r15665.
2007-07-27 13:33:20 +00:00
Josh Hursey
acbc8ecca3 - On Cray XT systems stop the grpcomm basic component from building.
grpcomm cnos component
  - Remove the .ompi_ignore
  - add a configure.m4 that should keep it from building on any system
    other than Cray XT* (copied from rml/cnos)
  - Fix some mis-named symbols resulting from cut/paste errors.

This patch brings the Cray build back into 'working' order.

This commit was SVN r15651.
2007-07-26 20:42:06 +00:00
Jeff Squyres
188d529beb * We *do* need the LSF task ID as part of our vpid
* Accidentally had the PLS LSF using the env SDS; switch it back to
   the LSF SDS

This commit was SVN r15650.
2007-07-26 20:22:36 +00:00
Josh Hursey
e5a03e7734 - Remove Makefile.in from version control
- Add back support for cnos (copy functionality lost by moving the interface
  from the RML).
- Fix some cut/paste errors.

This commit was SVN r15646.
2007-07-26 18:52:17 +00:00
Jeff Squyres
75192de1fc LSF support is now working. W00t! May be subject to a further tweak
or two.

 * checking lsb_init() is not sufficient to know whether you're in an
   LSF job or not; you also need to check for environment variable
   markers 
 * remove lots of debugging output
 * no need for the sds lsf to call lsb_init()
 * remove some slurm-like dead code and a copy-n-paste error in the
   sds lsf

This commit was SVN r15644.
2007-07-26 18:49:29 +00:00
Jeff Squyres
8e9c71282d Add a bunch more [conditional] debugging output.
This commit was SVN r15643.
2007-07-26 18:46:46 +00:00
Rich Graham
60df8be1a7 initial code - does not even compile, but Josh is picking up on this.
This commit was SVN r15641.
2007-07-26 17:55:51 +00:00
Brian Barrett
801fffabff Don't assume things about the contact info string in the general case. There
is no need for the IP address in most cases (filem being one dubious
exception), so just publish and hand around the supposedly opaque contact
info strings

This commit was SVN r15638.
2007-07-26 16:51:41 +00:00
Brian Barrett
e537cc0871 * Add documentation for RML base code
* Move function declaration out of base.h as it isn't needed
    outside the base code

This commit was SVN r15616.
2007-07-25 16:19:29 +00:00
Brian Barrett
f06b61cff9 Don't use the OOB TCP key for contact information, remove the need to
include a not so public header file.  FIxes a compile error on the Cray.

This commit was SVN r15613.
2007-07-25 15:12:07 +00:00
George Bosilca
00796cfdab Make sure the oob_tcp_windows_progress_callback is registered
in all cases. This is now done in the oob tcp open function.
As a result, the unregistering have to be done in the close
function.

This commit was SVN r15603.
2007-07-25 05:55:14 +00:00
George Bosilca
5d8a70e434 Update the Windows ODLS.
This commit was SVN r15600.
2007-07-25 03:57:25 +00:00