Here's the huge registry check-in you've all been waiting for with baited breath. The revised version sends a single message to all processes at the various stage gates, thus making the startup much more scalable. I could provide you with all the tawdry details, but won't for now - you are welcome to ask, though, and I'll merrily bore your ears to tears.
In addition, the commit contains the following:
1. set the ignore properties on ompi/debuggers and orte/mca/pls/poe
2. Added simplified subscribe and put functions to the registry's API. I have also converted all of the ompi functions that registered subscriptions to the new API, and caught their associated put's as well.
In a follow-on commit, I'll be adding support for George's hetero arch registry subscription (wanted to get this one in first).
This commit was SVN r7118.
Changed component so choice of decision functions controlled by mca params
(for now fixed decision functions (if statements) default)
started fixes for the various bcasts
This commit was SVN r7117.
(because we kept bumping up against the max filename limit in "tar"
when making tarballs, especially if the version number got long).
This commit was SVN r7065.
end) is to check the size and the extent (not the true extent).
If there are gaps we should adapt the source and the destination to match these gaps
(especially if the gaps are in the begining).
This commit was SVN r7060.
tree.
- fix up #include's throughout the tree (yay contrib/search_replace.pl!)
- remove a few extraneous #include's
- remove orte_sys_info*() from opal_init()/opal_finalize() (it's
already in orte_init_stage1() and orte_system_finalize())
- remove dependencies in opal on orte_system_info -- util/os_path.c
and util/os_create_dirpath.c (they only used path_sep, anyway --
easily changed to #defines)
This commit was SVN r7059.
- Change orte_base_infrastructre to orte_infrastructre to conform with
ompi_info's needs
- Move MCA Param registration in ORTE to a centralized function that is
called first in orte_init_stage1
- Set the infrastructre flag as an argument to orte_init
- Adjust initalization functions to properly pass down the infrastructre
flag.
This commit was SVN r7053.
mpi_kinds.ompi_module before the rest which depends on it. Fix suffix
rules, add suffix rule for libtool objects for testing. Fix for VPATH
builds.
This commit was SVN r7048.
API is still a bit unstable and may change.
- Add a primitive "first use" component that simply has each process
"touch" the pages that they want to use, thereby [hopefully] locking
them locally to a specific processor
- Add hooks in ompi_mpi_init to enable memory affinity when processor
affinity is used.
- Added hooks in ompi_mpi_finalize to shut down memory affinity when
it was initialized during ompi_mpi_init.
- Added right hooks in ompi_info to display maffinity components.
This commit was SVN r7044.
friendly #defines to be included in mpi.h (even for users), such as
_GNU_SOURCE, which can have some really big consequences on Linux.
Instead, add mpi.h to AC_CONFIG_HEADERS and just include the #defines
we have to have for mpi.h and the C++ bindings.
This commit was SVN r7022.
- Fix compiler warnings
- Fix problem with using "p" instead of "p_index"
- Style updates
- Check return of malloc() for NULL
This commit was SVN r6999.
- fix the --with-f90-max-array-dim configure switch
- fix configure test to find the supported f90 linker switch to find
fortran modules
- Unbelievably, some versions of sh (cough cough Solaris 9 cough
cough) actually seem to internally perform a "cd" into a
subdirectory when you run "./foo/bar", such that if you try to
source a script in the top-level directory in the bar script (i.e.,
". some_script" in the bar script), it will try to run it in the
"foo" subdirectory, rather than the top-level directory! #$@#$%#$%
So we have to pass in the pwd to the scripts so that they know where
some_script is.
- Reworked much of ompi/mpi/f90/Makefile.am for lots of reasons. See
the internal comments (mostly having to do with dependency stuff --
Libtool does not apparently support F90, so we can only build the
F90 library statically.
This commit was SVN r6993.
much time) and somewhat-lame implementation of barrier (need to
precompute some more stuff rather than calculate it every time).
Checkpointing so I can try this on another machine...
This commit was SVN r6985.
OPAL_ERROR, same for all the other error codes. Also, make sure that there
are never conflicts between OPAL anr ORTE error codes (for example).
Finally, provide opal_perror(), opal_strerror(), and opal_strerror_r() to
give stringified error messages for the different error codes
This commit was SVN r6969.
* Make ompi_info list timer components
* Remove flag to display whether we have memory intercepts (components are
already listed), until we can figure out how to do it *after* the
components are opened.
This commit was SVN r6950.
issue so PUT is default.. We are determining if this is an openib issue or a
btl issue as we have seen performance increases on mvapi.
This commit was SVN r6928.
to opal_progress() to use the timers instead of a tick count for deciding
whether to call the event loop or not. Currently supported platforms are:
- solaris (x86 / sparc)
- Linux (x86 / x86_64 / IA64)
- Mac OS X (x86 / Power PC)
This commit was SVN r6922.
that way anway) so that we can properly "make distcheck" easier. Fix
the rule for making the sym link to ptl_sm_send_alternate.c so that
"distcheck" works.
This commit was SVN r6919.
that this ORTE job is the only one on the nodes involved, and if
told what processors to assign the processes to, will bind MPI
processes to specific processors.
- Convert #include's to new style
- Convert some <tab>'s to spaces
This commit was SVN r6904.
* don't know what I was thinking, but can't use the MCA_PML_CALL macro on
the two data values, as they don't have things that the macro can
expand into
This commit was SVN r6868.
* Add base to memory framework so that we can do something sane with
ompi_info
* Updated ompi_info to print components for memory framework and
show whether we have memory hooks active or not.
This commit was SVN r6861.
of spaces (curses! indent(1) had been updated with a new option that
I did not use). This commit simply converts tabs to real spaces.
This commit was SVN r6799.
the values to the PML structure. This will allow PMLs that want to do
hardware matching at the cost of a smaller range of valid tags and cids.
Updated all the places that used the MPI_TAG_UB_VALUE constant to instead
look at the pml struct.
This commit was SVN r6778.
alignment of 0, then assume there will be no data segment and don't do
the checks to see if it will be beyond the end of the file.
This commit was SVN r6773.
The following SVN revision numbers were found above:
r6672 --> open-mpi/ompi@8b56769307
- Adjust btl sm to allocate just a few bytes extra to allow the common
sm component to assume that there will be a data segment (even though
the sm btl doesn't use the data segment in that portion of code)
This commit was SVN r6772.
* Major rework of Portals to better match Red Storm and hopefully get
better performance:
- Always assume there is only one module (since there are no machines
on the planet with more than one Portals interface)
- make progress all one function rather than dispatching to other
functions and dispatch on event type, not comm type
- remove polling of unneeded events
This commit was SVN r6769.
that were set on the command line. This was techinically exactly the
way the code was designed, but it certainly violated the Law of Least
Astonishment (even to its designer ;-) ). So now if you execute
something like this:
mpirun -mca pls_rsh_debug 1 -np 4 hello
You'll see debugging output from the rsh pls component, as you would
expect (this was not previously the case -- the MCA pls_rsh_debug
parame would be set to 1 in the 4 spawned hello processes, but *not*
in the orterun process).
More specifically, MCA parameters will be set in the orterun process
in the following cases:
- The new command line switch "--gmca" (or "-gmca") is used,
indicating that the MCA parameter is "global". --gmca also means
that that MCA parameter will be applied to all context app's. For
example:
mpirun -gmca foo bar -np 1 hello : -np 2 goodbye
The foo MCA param will be set in both the hello and goodbye
processes.
- If there is only one context app. For example:
mpirun -mca pls_rsh_debug 1 -np 4 hello
will set pls_rsh_debug to 1 in both the orterun process and the 4
spawned hello processes.
Also added a few more comments inside orterun to document a somewhat
confusing use of a state variable in a recursive case.
This commit was SVN r6764.
Change all the places where they are used to fit the new name.
Remove the code to check the remote arch from the PML. We will have a GPR mechanism
in ompi_mpi_initialize to do that.
This commit was SVN r6750.
the message is no longer pending
* Try to push out new messages whenever we finish a send, whether it
worked or not. Means that in the case where the other side has too
many sends pending, we'll constantly retry one (and only one, once the
pending number is reached) message until goodness returns
* Make some warnings only happen in verbose case, as they are mainly
diagnostics
This commit was SVN r6732.
to a tty or not. Now you can do something like:
ompi_info -all | grep btl_portals
and get the full line for each btl_portals parameter.
* For the case where stdout is a tty, we have my current nomination for
Today's Useless OMPI Feature. Autodetect the width of the terminal, so
people with really wide terminals will get less wrapping
This commit was SVN r6722.
may appear.
(remove *error.h file from Makefile.am -- a cut-n-paste error that has
propagated to a surprising number of directories ;-) )
This commit was SVN r6721.
- new preferred API calls for registering MCA parameters are
mca_base_param_reg_{int|string} and
mca_base_param_reg_{int|string}_name.
- See opal/mca/base/mca_base_param.h for docs on new calls.
- Can now register and lookup a value at the same time.
- Can now mark a parameter "read only" at registration time
- Can now mark a parameter "internal" at registration time
- Can now associate a help message with the parameter at registration
time; displayed in the ompi_info output.
The old API calls are still available for backwards compatibility
(mca_base_param_register_{int|string}. They will eventually be
removed -- all developers are encouraged to use the new APIs from here
on out and replace any old calls with the new API.
Some params were also renamed -- the previous convention of using
"base_" as a prefix for any param that was not associated with a
component is henceforth deprecated. Instead, use one of the following
prefixes:
mca: for anything in the MCA base itself
opal: for anything in OPAL
orte: for anything in ORTE
mpi: for anything in OMPI
This commit was SVN r6698.
This required a little fiddling with a number of areas. Biggest problem was that it uncovered a potential for an infinite loop to be created in the registry. If a callback function modified the registry, the registry checked the triggers to see if anything had fired. Well, if the original callback was due to a trigger firing, that condition hadn't changed - so the trigger fired again....which caused the callback to be called, which modified the registry, which checked the triggers, etc. etc.
Triggers are now checked and then "flagged" as being "in process" so that the registry will NOT recheck that trigger until all callbacks have been processed. Tried doing this with subscriptions as well, but that caused a problem - when we release processes from a stagegate, they (at the moment) immediately place data on the registry that should cause a subscription to fire. Unfortunately, the system will just hang if that subscription doesn't get processed. So, I have left the subscription system alone - any callback function that modifies the registry in a fashion that will fire a subscription will indeed fire that subscription. We'll have to see if this causes problems - it shouldn't, but a careless user could lock things up if the callback generates a callback to itself.
Also fixed the code that placed a process' RML contact info on the registry to eliminate the leading '/' from the string.
This commit was SVN r6684.
* add btl back into ompi_info. Since it now directly calls the
open/close, the missing symbol problems Ralph was seeing when ob1 is
ignored will not occur.
This commit was SVN r6652.
support in OMPI. Currently only enables/disables the architecture
sharing modex in ob1 pml.
* Add sds framework to ompi_info
* Figure out table ids to use for Portals BTL at configure time, since
we should use 30 & 31 on Red Storm, but the reference implementation
only supports 0-8.
* Some bug fixes in Portals UTCP sds
This commit was SVN r6650.
- only call sched_yield if it exists
- don't fail out if modex doens't work in ob1
- bunch of fixes for Portals BTL
- add cnos rml component
- add NULL gpr component (should only be used if replica AND proxy
fail to load)
This commit was SVN r6629.
Any non-blocking P2P function should set the request to something else
then MPI_REQUEST_NULL, when passed a source/dest of MPI_PROC_NULL.
Introduce a new ompi_request_empty, which returns the
MPI_SOURCE=MPI_PROC_NULL, MPI_TAG=MPI_ANY_TAG, count=0 as specified by
the MPI-standard.
This commit was SVN r6611.
* remove the recv fragment code, since it really isn't needed
* handle memory descriptor binding a bit more sanely, and use
thresholds so that Portals does the unlink for us, when it feels
like it.
This commit was SVN r6575.
* change all the opal_output_verbose calls in the critical path to
OPAL_OUTPUT_VERBOSE so that they are pre-processed out if debugging
is not enabled
* remove stub code
This commit was SVN r6564.
"run-time" api for the reference implementation.
* Make the non-modex utcp and redstorm compat code do the same things in
the same order
This commit was SVN r6556.
associated with the descriptor is ours again once the callback function
returns. Make it so - probably can optimize out some of the stuff I
did when I mistakenly thought the descriptor free() was called on the
passed descriptor
* Fix some dumb accounting errors with MD usage for unexpected receives
This commit was SVN r6555.
Fixed receive descriptor counts that limited mvapi and openib to 2 procs.
Begin porting error messages to use the BTL_ERROR macro.
This commit was SVN r6554.
1. Modify the registry to eliminate redundant data copying for startup messages.
2. Revise the subscription/trigger system to avoid redundant storage of triggers and subscriptions. This dramatically reduces the search time when a registry action occurs - to illustrate the point, there are now only a handful of triggers on the system for each job. Before, there were a handful of triggers for each PROCESS in the job, all of which had to be checked every time something happened on the registry. This is much, much faster now.
3. Update all subscriptions to the new format. There are now "named" subscriptions - this allows you to "name" a subscription that all the processes will be using. The first one to hit the registry actually defines the subscription. From then on, any subsequent "subscribes" to the same name just cause that process to "attach" to the existing subscription. This keeps the number of subscriptions being tracked by the registry to a minimum, while ensuring that each process still gets notified.
4. Do the same for triggers.
Also fixed a duplicate subscription problem that was causing people to receive data equal to the number of processes times the data they should have received from a trigger/subscription. Sorry about that... :-( ...but it's all better now!
Uncovered a situation where the modex data seems to be getting entered on the registry a second time - the latter time coming after the compound command has been "fired", thereby causing all the subscriptions to fire. Asked Tim and Jeff to look into this.
Second phase of the changes will involve modifying the xcast system so that the same message gets sent to all processes. This will further reduce the message traffic, and - once we have a true "broadcast" version of xcast - really speed things up and improve scalability.
This commit was SVN r6542.
- we modex send and receive a structure containing the nid id and the endpoint id. On the
remote node we can recompose the endpoint_addr via mx_connect.
- accept several retry to mx_connect (up to 5 seconds ... soon to be a MCA param).
- correctly construct/destruct the internal objects.
- some others minor changes.
This commit was SVN r6535.
multiple components to share a single mpool module (e.g., the
ptl/btl and coll sm components).
- Re-tool the ptl, btl, and coll sm components to first look for the
target mpool module, and if they don't find it, to create it.
- coll sm component now correctly identifies when it is supposed to
run or not (i.e., if all the processes in the communicator are on
the same host). Now we just need to fill in some algorithms. :-)
This commit was SVN r6530.
corrected free and realloc in mpool. Added alloc_base to
mca_mpool_base_registration_t to be used as the actual alloc'd base address,
which may be different from the reported base address due to page allignment.
This commit was SVN r6524.
IBM tests, but see same segfaults / assets in sm btl.
* Add prepare_src implementation so that we can send multiple fragments of
large messages
* Add queuing of sends if either there are too many outstanding sends
(we have to limit this so that we don't have more sends pending than
we could get acks for) or if we get an ack with a 0 byte mlength,
which means the remote side dropped the message on us.
Still need to valgrind to make sure I'm not leaking resources
This commit was SVN r6508.
- Change ompi_proc_world() to only return the procs in this job (as
opposed to all of them)
- Add a subscription that fires during MPI_INIT (stg1) for figuring
out which procs are on my local node. Need to figure out what to do
in the esoteric cases -- but the obvious one (Red Storm), where
subscriptions are never fired, is ok, because by definition, no
other procs will be on my node, so their default value (not on my
node) is ok.
--> Need to have RHC check this code; it seems to work, but I think
I'm getting too much data back from the subscription.
- End result is that any proc that is on my node will have its
OMPI_PROC_FLAG_LOCAL bit set on its proc->proc_flags field.
- Added/corrected a few comments in proc.h.
This commit was SVN r6507.
think the data is contiguous - and the convertor routine
you've changed this to doesn't support returning the correct
offset into the user buffer when a NULL address is provided
in the iovec array
This commit was SVN r6496.
* Add ability to completely disable libltdl (the dlopen code to load
dynamic shared objects) to configure: --disable-dlopen
* Added MCA param (component_disable_dlopen) to disable DSO loading
at runtime
* Made the event library behave in some not-completely-erroneous way
on platforms where it has absolutely no eventops support (ie, no
select, poll, or epoll)
* Disabled orte_wait, opal_few, and opal_daemon_init code on
platforms without fork, waitpid support. All non-init functions
will return OPMI_ERR_NOT_SUPPORTED
* Disable orteprobe tool when fork or pipe aren't supported
This commit was SVN r6490.
Red Storm. Add stub functions to ompi_config_bottom.h when they are
around
* Add protection for a bunch of #include <netinet/in.h>s
* Fix up the Portals BTL so that it compiles on Red Storm and has the
right mojo for initialization on Red Storm
* Add some important comments to ompi_check_package and mvapi configures
* Add support for platforms without getpwuid() (aka, Red Storm).
This commit was SVN r6478.
all the time. There is a performance problem (it's a lot slower than the optimized versions)
but otherwise it will never get tested intensively.
This commit was SVN r6467.
sockaddr_in - seems to be a good indicator)
* disable util/if code if no inet devices (again, no sockaddr_in)
* add enable/disable flag to disable stacktrace pretty-print code
(defaults to enabled). Seems there's something funky going on with
the preprocessor on Red Storm that was causing problems - this was
the easiest fix
* clean up a bunch of the configure.m4 files to remove bogus comments,
properly comment them, fix the dumb logic for happy/unhappy
* Create a macro for testing both header and library for a package,
since we seem to do this kind of test quite often. Handles the
-I and -L search paths properly (including stripping out /usr and
/usr/local if not needed)
* Converted mvapi components to configure.m4, using the nice new
ompi_check_package macro (above)
This commit was SVN r6454.
threads (basically, same as before, but we now link the right thread
libraries).
* Add disable-io-romio flag to disable compiling ROMIO
* Migrathe mvapi btl from configure.stub to configure.m4
This commit was SVN r6453.
was that some datatypes can be used in order to create additional datatypes. In such cases
they should not get destroyed otherwise the user will not be able to retrieve how a
datatype was created. So I decide to never increase the reference count for any predefined
datatypes (as we already know they will never get destroyed, except on finalize). For
the others, every time a datatype is user by another one I increase the reference count.
When I destroy a datatype, I parse (the internal structure args) and release one ref count
on all not predefined datatypes used by this one. Thus the datatypes get cleaned. The main
problem with this approach is the recursivity as this function can trigger another call
to itself (but I dont think it will be an issue).
This commit was SVN r6450.
appropriate spot and also added a check for the libsysfs library required by openmpi. Modified the mvapi configure.stub to use AC_TRY_LINK for
pthreads.
This commit was SVN r6441.
(which it is) and not romio itself
* Work around dark, evil linker voodoo that prevented building both shared
and static libraries on OS X. The global variables in ad_init.c were
not initialized, and so were marked as tentative definitions, which caused
much pain for the linker later on. Initializing them makes them
actual definitions and the problem goes away. I hate linkers.
This commit was SVN r6439.
present then all PTL's not in the include list are marked as not available. If the exclude
list is present then all PTLs in this list are marked as not available.
This commit was SVN r6438.
is a lot more difficult than a PTL, and it can adapt it's behavior to the level of threading required
by the user. In this case the behavior is the priorit of the PML. Therefore this information is never
availale before the init function (of the PML) is called. So I try to keep nearly the same structure
as it was before, with one change. When a PML get initialized it does not necessarily means it has been
selected, so it does not means it has to create all it's internal structures (and select the PTL and
all this stuff). They can all be done later, when a PML knows that it definitively get selected
(when the enable function is called with the argument set to true). Thus, in the case of a PML close
one have to check if the PML has been selected or not before trying to clean up the internals.
I had to change the MPI_Init function to allow the PML to be enabled before we start adding procs inside.
This commit was SVN r6434.
all required attributes are set even if the trigger never fires (the
trigger can update the value, if needed. This is all in MPI_Init, so
that does not violate the standard)
This commit was SVN r6427.
components to succeed with --enable-dist. Instead, just add them to
all_components and make dist will still work - we're going to stamp out
the Makefiles no matter what
* Add missing header to ob1 pml for make dist
* Clean up the Portals BTL configure code
This commit was SVN r6413.
frameworks, and components without configure scripts instead of
hard-coded shell variables (for projects and frameworks) and
shell variable building (for components).
* Add 3rd category of component configuration (in addition to configure
scripts and no-configured components): configure.m4 components. These
components can only be built as part of OMPI (like no-configure), but
can provide an m4 file that is run as part of the main configure
script. These macros can set whether the component should be built,
along with just about any other configuration wanted. More care must
be taken compared to configure components, as doing things like setting
variables or calling AC_MSG_ERROR now affects the top-level configure
script (so calling AC_MSG_ERROR if your component can't configure
probably isn't what you want)
* Added support to autogen.sh for the configure.m4-style components,
as well as building up the m4_define lists ompi_mca.m4 now expects
* Updated a number of macros to be more config.cache friendly (both
so that config.cache can be used and so the test can be quickly
run multiple times in the same configrue script):
- ompi_config_asm
- c_weak_symbols
- c_get_alignment
* Added new macros to be shared when configuring components:
- ompi_objc.m4 (this actually provides AC_PROG_OBJC - don't ask...)
- ompi_check_xgrid
- ompi_check_tm
- ompi_check_bproc
* Updated a number of components to use configure.m4 instead of
configure.stub
- btl portals
- io romio
- tm ras and pls
- bjs, lsf_bproc ras and bproc_seed pls
- xgrid ras and pls
- null iof (used by tm)
This commit was SVN r6412.
* mpi_show_mca_params
If set to true, this turns on the dumping of all MCA parameters when MPI_INIT is called.
Only the 'rank 0' processes will print the parameters.
* mpi_show_mca_params_file
(This value is only used if the first argument is set to true) If this value is non-NULL
it specifies the file to put the dump into. This file can then be used as input to mpirun
for debugging purposes. If this value is not set (and mpi_show_mca_params is set) then
the parameters are dumped to stdout.
This commit was SVN r6401.
the external32 conversions (as specified in the MPI standard) the other one is the local
convertor that can be used for MPI_Pack and MPI_Unpack functions.
This commit was SVN r6375.
- After long discussions and ruminations on how we run components in
LAM/MPI, made the decision that, by default, all components included
in Open MPI will use the version number of their parent project
(i.e., OMPI or ORTE). They are certaint free to use a different
number, but this simplification makes the common cases easy:
- components are only released when the parent project is released
- it is easy (trivial?) to distinguish which version component goes
with with version of the parent project
- removed all autogen/configure code for templating the version .h
file in components
- made all ORTE components use ORTE_*_VERSION for version numbers
- made all OMPI components use OMPI_*_VERSION for version numbers
- removed all VERSION files from components
- configure now displays OPAL, ORTE, and OMPI version numbers
- ditto for ompi_info
- right now, faking it -- OPAL and ORTE and OMPI will always have the
same version number (i.e., they all come from the same top-level
VERSION file). But this paves the way for the Great Configure
Reorganization, where, among other things, each project will have
its own version number.
So all in all, we went from a boatload of version numbers to
[effectively] three. That's pretty good. :-)
This commit was SVN r6344.
* rename ompi_malloc to opal_malloc
* rename ompi_numtostr to opal_numtostr
* start of rename of ompi_environ to opal_environ
This commit was SVN r6332.
* rename ompi_basename to opal_basename
* rename ompi bitop functions to opal
* rename ompi_cmd_line to opal_cmd_line
* rename ompi_sizet2int to opal_sizet2int
* rename orte_daemon_init to opal_daemon_init
* rename ompi_few to opal_few
This commit was SVN r6330.
- move mpool and allocator frameworks back to ompi (from opal)
- specialize the ompi_free_list class to use an mpool instance
- un-specialize opal_free_list to *not* use mpool; just use malloc/free
This commit was SVN r6292.
