and another for dst descriptors. This provide partial solution to OB1 protocol
deadlock problem. We can limit number of RDMA descriptors (by setting
btl_openib_free_list_max to something different from -1) and if we will be
lucky to hit this limit before we fail to register more memory the protocol
will not deadlock. When we had only one list for src/dst descriptors we
deadlocked when we reached max limit for the list.
This commit was SVN r13844.
Algorithm allows user to specify the segment size to be used for computation/communication overlap.
The additional memory requirement for the algorithm is 2 x segment size.
It performed well for (really) large message sizes over MX and it passed intel Allreduce_c and Allreduce_loc_c tests.
This commit was SVN r13832.
Long unexpected messages were not generating PUT_START events
because the MD for long unexpected messages was configured to
ignore start events. When a long unexpected message arrived, it
traversed the match list, and ended up in the long unexpected MD.
As the long message is being consumed, the code called PtlMDUpdate()
to look for the message, but there was no event that indicated
that it had arrived. So, the update succeeded. Once the long
unexpected message was consumed, the PUT_END event showed up in the
event queue -- except the code wasn't looking for it anymore.
The PUT_START events exist specifically to handle ordering between
short and long unexpected messages, so PUT_START events can't be
ignored on long unexpected messages.
Modified the code to generate PUT_START events for both long and
short unexpected messages and handle matching up START and END
events appropriately.
This commit was SVN r13746.
- the block sizes are computed in more uniformn way.
The first k blocks may be 1 element larger than the remaining blocks.
The algorithm passed Intel Allreduce_c and Allreduce_loc_c tests, and
IMB-3.2 Allreduce, over TCP and both btl and mtl MX (up to 128 processes).
The algorithm still only supports commutative operations.
This commit was SVN r13738.
- Implement the BML/r2 finialize funciton
- Cleanup the btl close routine
- Wire up a pml_base_verbose MCA parameter so you can actually watch the PML selection logic if you really want to.
- Fix a potental segfault in the selection logic.
ompi_pointer_array_get_item() may return NULL, so we have to check for it
This commit was SVN r13734.
The following SVN revision numbers were found above:
r2 --> open-mpi/ompi@58fdc18855
outstanding requests can be limited using mca parameters.
The implementation passed Intel, IMB-3.2, and mpi_test_suite tests over
TCP and MX up to 128 processes (64 nodes), on both 32-bit and 64-bit machines.
It is not activated by default, but it should be useful for really large
communicator sizes.
This commit was SVN r13720.
middle of a struct). Now we properly define and name the union
outside the struct and simply create an instance of it inside the
struct.
This commit was SVN r13709.
support SEND_IN_PLACE causes badness because the BTL tries to use the
not-exactly-complete convertor. Don't need it in this situation anyway.
This commit was SVN r13700.
Implementation passed intel: MPI_Reduce_c , MPI_Reduce_loc_c, and MPI_Reduce_user_c tests
over TCP, BTL MX, and MTL MX, as well as, mpi_test_suite Reduce tests (up to 64 nodes).
The algorithm is still not activated by decision function (will be in the near future).
This commit was SVN r13657.
when on the Cray machine (aka when the NULL GPR is in use).
This commit was SVN r13638.
The following SVN revision numbers were found above:
r13582 --> open-mpi/ompi@041beeb1b6
buffer fails. If cb is already allocated, but it is full and allocation of
additional cb fails, we spin waiting for receiver to free space in existing
cb.
This commit was SVN r13635.
* Update exactly how we handle MPI exceptions, particularly with
respect to MPI-1 section 3.2.5, and how error handlers are only
invoked for the ''first'' request that generates an exception.
* Update the "see also" section to be consistent across all 8
MPI_Test* and MPI_Wait* functions.
* Fixes trac:560
This commit was SVN r13619.
The following Trac tickets were found above:
Ticket 560 --> https://svn.open-mpi.org/trac/ompi/ticket/560
adding new procs that the remote proc's pml is the same as our local pml.
Turns the hangs from mismatched PMLs into an abort, which is better,
I think.
This commit was SVN r13582.
open MTL MX and BTL MX and initialize them at the same time. The problem is
that both call mx_init and mx_finalize, solution is to add an external entity
that does the init and finalize (based on ref counting).
This commit was SVN r13576.
The C++ bindings were not tracking keyvals properly -- they were
freeing some internal meta data when Free_keyval() was called, not
when the keyval was actually destroyed (keyvals are refcounted in the
C layer, just like all other MPI objects, because they can live for
long after their corresponding Free call is invoked). This commit
fixes this problem and several other things:
* Add infrastructure on the ompi_attribute_keyval_t for an "extra"
destructor pointer that will be invoked during the "real"
constructor (i.e., when OBJ_RELEASE puts the refcount to 0). This
allows calling back into the C++ layer to release meta data
associated with the keyval.
* Adjust all cases where keyvals are created to pass in relevant
destructors (NULL or the C++ destructor).
* Do essentially the same for MPI::Comm, MPI::Win, and MPI:Datatype:
* Move several functions out of the .cc file into the _inln.h file
since they no longer require locks
* Make the 4 Create_keyval() functions call a common back-end
keyval creation function that does the Right Thing depending on
whether C or C++ function pointers were used for the keyval
functions. The back-end function does not call the corresponding
C MPI_*_create_keyval function, but rather does the work itself
so that it can associate a "destructor" callback for the C++
bindings for when the keyval is actually destroyed.
* Change a few type names to be more indicative of what they are
(mostly dealing with keyvals [not "keys"]).
* Add the 3 missing bindings for MPI::Comm::Create_keyval().
* Remove MPI::Comm::comm_map (and associated types) because it's no
longer necessary in the intercepts -- it was a by-product of being
a portable C++ bindings layer. Now we can just query the C layer
directly to figure out what type a communicator is. This solves
some logistics / callback issues, too.
* Rename several types, variables, and fix many comments in the
back-end C attribute implementation to make the names really
reflect what they are (keyvals vs. attributes). The previous names
heavily overloaded the name "key" and were ''extremely''
confusing.
This commit was SVN r13565.
The following Trac tickets were found above:
Ticket 817 --> https://svn.open-mpi.org/trac/ompi/ticket/817
investivating #817:
* Remove use of legal_numbits member and always just use the full
size of the array. There was a corner case where legal_numbits was
not an even multiple of the number of bits in the array where bits
would not get freed properly, ususally causing wasted fortran
MPI handles, or, as in the case of #817, wasted attribute keyvals
(i.e., the user freed them, but the bitmap didn't reflect the
free).
* Re-order some error checks to ensure that we don't segv (we don't
currently trigger this problem anywhere; I just noticed it while
doing the other attribute keyval and legal_numbits work).
Since this change affects all Fortran MPI handles, I ran all the intel
and ibm tests and all still pass with this change.
This commit was SVN r13561.
Add new function opal_get_num_processors() that will return the number
of processors on the local host. Does the Right thing in POSIX
environments (to include a special case for OS X), and will shortly do
the Right Thing for Windows (this commit includes a change to
configure, so I wanted to get that in before the US workday -- the
Windows code can some shortly because it won't involve configury
changes).
This commit was SVN r13506.
The following Trac tickets were found above:
Ticket 853 --> https://svn.open-mpi.org/trac/ompi/ticket/853
a protection against creating subarrays with MPI_LB and MPI_UB. But, I don't
think the MPI standard state anything about this.
This commit was SVN r13504.
The step used to iterate through buffer was function of true_extent instead of extent.
This may or may not solve ticket #689 because I am still getting failures over btl mx,
but I cannot reproduce failures over mtl mx nor tcp.
This commit was SVN r13459.
1. if the user has specified sched_yield, we simply do what we are told
2. if they didn't specify anything, try to get the number of processors on this node. Note that we already now get the number of local procs in our job that are sharing this node - that now comes in through the proc callback and is stored in the ompi_proc_t structures.
3. if we can get the number of processors, compare that to the number of local procs from my job that are sharing my node. If the number of local procs exceeds the number of processors, then set sched_yield to true. If not, then be a hog and set sched_yield to false
4. if we can't get the number of processors, default to conservative behavior and set sched_yield to true.
Note that I have not yet dealt with the need to dynamically adjust this setting as more processes are added via comm_spawn. So far, we are *only* looking within our own job. Given that we have now moved this logic to mpi_init (and away from the orteds), it isn't yet clear to me how a process will be informed about the number of procs in *other* jobs that are also sharing this node.
Something to continue to ponder.
This commit was SVN r13430.
* The real fix, don't leave the OOB in blocking mode during comm_dyn_init(),
as it means no progressing MPI events while the event library is waiting
for TCP stuff to come in.
* Add many comments explaining the reasons for the current ordering
This commit was SVN r13422.
* have the mpool size be based on MCW, not num procs
in other jobs we know about. Solves the problem of
the spawned job having a much bigger than needed
sm file
* Can't assume that "me" is in the list of procs
passed to addprocs, so need to use slightly different
logic and not go through all of add procs unless
there's a proc in my job that isn't me.
This seems to greatly improve the situation, although
there still seems to be more of a slowdown through
MPI_INIT for the children (if there are more than one
child) than MPI_INIT for the parent if there are 'n'
children compared to 'n' parents. Hopefully that
made sense ;)
This commit was SVN r13417.
of the first entry might not be the start of the user's buffer. This is
similar to what ompi_convertor_unpack does. This is the solution for
the test case attached to ticket #690.
Refs trac:690
This commit was SVN r13397.
The following Trac tickets were found above:
Ticket 690 --> https://svn.open-mpi.org/trac/ompi/ticket/690
not being able to take C function pointers for either of the
copy or the delete fn. Fix by overloading the Create_keyval methods.
Fix trac #737, #738. Reviewed by jsquyres.
* A couple of cxx tests in ompi-tests (winkeyval.cc & typekeyval.cc)
will be re-enabled to regression test for this fix.
This commit was SVN r13391.
completed successfully, Bad Things(tm) could happen.
* Now we explicitly check orte_initialized (a new global in ORTE
indicating whether we are between orte_init() and orte_finalize()
or not), and if so, react accordingly.
* If ORTE is initialized, use orte_system_info.nodename; otherwise,
use gethostname().
* Add loop protection to ensure that ompi_mpi_abort() is not invoked
multiple times recursively.
This commit was SVN r13354.
know what my local rank is, and therefore set my paffinity ID as
appropriate. Specifically, we're no longer relying on the
special/secret mpi_paffinity_processor MCA parameter that the orted
would set for us.
This allows processor affinity to be used in environments where the
orted is not used (e.g., bproc, and someday in the hopefully not
too-distant future, SLURM).
This commit was SVN r13352.
The following SVN revision numbers were found above:
r13351 --> open-mpi/ompi@a338b7e533
Over to Jeff now for modifying mpi_init accordingly.
Until Jeff makes his changes, nobody should see anything different as the new info just isn't used by anything!
This commit was SVN r13351.
MPICH2 for "small" commutative operations in the reduce_scatter basic
implementation. "small" is currently pretty big, as it doesn't take
much to beat reduce/scatterv. Need to do much more than this for
better all around performance of MPI_Reduce_scatter, but this was enough
to solve the problems I was having.
This commit was SVN r13348.
Found another places that we were incorrectly casting a C++ MPI handle
array to the corresponding C array type and hoping for the best (which
won't work at all). This commit fixes things so that we now do the
proper conversion between C<-->C++ handles.
This commit was SVN r13346.
and George on these refinements):
* Rename the static OBJ initializer macro to be
OPAL_OBJ_STATIC_INIT(class)
* Ensure that all static OBJ initializations get a refcount of 1
(doesn't ''really'' matter, since they're static, it should never
get to the point where the OBJ is DESTRUCTed, but more correct
nonetheless)
* Add a "magic number" to the OBJ when compiling with debug support.
The magic number does some rudimentary support to ensure that
you're operating on a valid OBJ (and fails an assertion if you're
not). Check to ensure that the memory contains the magic number
when performing actions of OBJ's. Also remove the magic number
when DESTRUCTing OBJs, so that if, for example, an OBJ is
DESTRUCTed more than once, we'll fail the magic number assert.
This commit was SVN r13338.
The following SVN revision numbers were found above:
r13227 --> open-mpi/ompi@96030de97b
r13228 --> open-mpi/ompi@c2e9075d29
- post isends in reverse order of posting irecvs.
if the messages arrive approximately in order, this should
minimize the time spent in matching the requests.
I did not see any performance difference over MX up to 64 nodes, but
the change makes sense and may have some impact when we have (many)
more nodes.
This commit was SVN r13337.
configured with --disable-mpi-cxx so that the default -I flags in the
wrapper compilers don't point to a directory that doesn't exist.
Thanks to Martin Audet for identifying the problem.
This commit was SVN r13296.
- Allreduce algorithms:
- Recursive doubling is used for small messages (up to 10KB) and can be used for
both commutative and non-commutative operations.
Recursive doubling passed OCC, IMB-3.2, Intel (Allreduce_c, Allreduce_loc_c, and
Allreduce_user_c), mpi_test_suite (Allreduce MIN/MAX, and Allreduce MIN/MAX with
MPI_IN_PLACE) tests on TCP up to 36 nodes and MX up to 64 nodes.
- Ring algorithms performs well for larger messages but cannot be used for
non-commutative operations. It passed the same tests as recursive doubling, except
some of the non-commutative tests in Intel benchmarks Allreduce_loc_c and Allreduce_user_c
(which was expected).
- MPI_Allreduce with new decision function passed all of the tests mentioned above.
- Cleaning up coll_tuned_util. Moving isendrecv to static inline just like sendrecv.
This commit was SVN r13252.
not the component. This potentially allows for a mix of HCAs that
support eager RDMA and those who do not on a port-by-port basis.
This commit was SVN r13242.
* If the text to cite where the problem occurred is "\n", prettyprint
somethign a little nicer so that it's clear that we're talking
about the end of line
* Add a missing help message ("ini file:unknown field"), and display
it a little better (i.e., show the erroneous field, not a
misleading "end of line" marker)
* It's "OpenIB", not "Open IB"
This commit was SVN r13241.
needlessly registered in multiple different places, and none of them
had a good help string. There was also an inconsistent check for
setting both mpi_leave_pinned and mpi_leave_pinned_pipeline (i.e., it
was only in ob1). This commit moves the registration of these params
to one central place (ompi/runtime/ompi_mpi_params.c, with all other
mpi_* MCA params) and uses globals to propagate the values as
relevant. The error check was also moved to the central location to
ensure that we can consistency everywhere.
This commit was SVN r13226.
