interconnects that provide matching logic in the library.
Currently includes support for MX and some support for
Portals
* Fix overuse of proc_pml pointer on the ompi_proc structuer,
splitting into proc_pml for pml data and proc_bml for
the BML endpoint data
* bug fixes in bsend init code, which wasn't being used by
the OB1 or DR PMLs...
This commit was SVN r10642.
yes this means it WAS possible for two nodes to choice two different algorithms
(discovered by Doug Gregor and figured out by George)
Also changed some names like size to comsize so we know which sizes we are using where
This should be updated in al versions
This commit was SVN r10601.
order the BTL depending on the real latency for the eager protocol. Starting from now, the
latency one can specify for the devices will be in micro-second, while the bandwidth is in Mbs
(as it was before).
This commit was SVN r10566.
mpi_leave_pinned when multiple OpenIB HCA ports are found.
Specifically, if mpi_leave_pinned == 1 and ultiple HCA ports are
found, the MCA parameter btl_openib_max_btls is set to 1. If the MCA
parameter btl_openib_warn_leave_pinned_multi_port is true, emit a
warning that this happened (having an MCA parameter to control the
warning allows users/sysadmins to turn it off instead of being nagged
for every run).
This commit was SVN r10521.
explicitly enabled at run-time with the mca parameter
io_romio_enable_parallel_optimizations set to something non-zero.
This will enable some magic flags in Panasas if the user didn't
set them (either on or off) and do some slightly better things
with strided collective writes.
This commit was SVN r10516.
standard). This macro allow us to specify the length of the fragment. Now we are
able to know how the message is fragmented between the network devices or inside
the communication protocol.
This commit was SVN r10508.
specified check that the put function is available for the BTL. Same safe check for
the GET function. At the end make sure that at least on communication protocol is
specified, otherwise force the send flag.
This commit was SVN r10507.
by the BTL (btl_max_rdma_size). Now the PUT protocol is pipelined even if there
is just one network between the 2 peers. Unfortunately, this problem is present
the 1.1 (no pipeline for the PUT protocol).
This commit was SVN r10499.
time.
UD is connectionless, and as long as peers are statically assigned to QPs,
there is no reason to set up the adressing information lazily.
Lots of code was axed, as endpoints no longer have state. Removed a
number of other elements in the endpoint struct to make it as lightweight
as possible.
I was able to remove an entire function call/branch in the send path,
which I believe is the main contributor to a 2us drop in NetPIPE latency.
Some whitespace cleanups as well.
Passes IBM test suite, and all but certain intel tests that were failing
before the change, over ob1 PML.
This commit was SVN r10494.
Moved a lot of the module-specific init from the component init to the module init.
Try keeping a pointer to reduce indexing, didn't seem to help - leaving in place
for now.
This commit was SVN r10485.
Playing around with OPAL_LIKELY/UNLIKELY, no real gains yet.
Reworked progress() to process many WC's at a time, as well
as immediately repost groups of receive buffers.
This commit was SVN r10481.
1. ompi/mca/btl/udapl/btl_udapl_proc.c should be including
btl_udapl_endpoint.h for mca_btl_udapl_proc_insert function.
2. btl_udapl_endpoint.c it looks like you are using
&endpoint->endpoint_lock when you should use &ep->endpoint_lock in a
OPAL_THREAD_LOCK call.
3. btl_udapl_frag.h has a couple opal_list_item_t's that should be
ompi_free_list_item_t in the _FRAG_ALLOC_{EAGER,MAX} macros.
This commit was SVN r10442.
call to the _UNPACK macro, in the case where the length of the received data
is zero. This might happens on the PUT protocol.
This commit was SVN r10431.
mpi_leave_pinned when multiple OpenIB HCA ports are found.
Specifically, if mpi_leave_pinned == 1 and ultiple HCA ports are
found, the MCA parameter btl_openib_max_btls is set to 1. If the MCA
parameter btl_openib_warn_leave_pinned_multi_port is true, emit a
warning that this happened (having an MCA parameter to control the
warning allows users/sysadmins to turn it off instead of being nagged
for every run).
This commit was SVN r10424.
send/receive queues if there is something available in the short message
rdma queues, then we have to poll *ALL* the rdma queues before exiting,
or we aren't fair about frag reception and fall into degenerate matching
cases.
This commit was SVN r10410.
where to look for registrations that were used in the alloc/free code don't
work (because the memory returned from malloc() -- whowever gets around to
calling it) might actually be registered already. So just call malloc
and free directly and avoid the whole issue when leave pinned is on. After
all, you have to pay the registration cost sometime, and if leave pinned
is on, you only have to pay it once. It makes things much simpler to
have that once be at first use rather than during ALLOC_MEM, and as far
as I can read, we're still standards conformant this way.
This commit was SVN r10406.
- Added some basic flow control to limit number of posted sends.
- Merged endpoint send/recv lock into single endpoint lock.
- Set the LMR triplet length in the send path, not at allocation time.
This has to be done because upper layers might send less than the
amount allocated.
- Alter the tie-breaker if statement protecting the second call
to dat_ep_connect(). The logic was reversed compared to the tie-
breaker for the first dat_ep_connect(), making it possible for
3 or more processes to form a deadlock loop.
- Some asserts were added for debugging purposes.. leaving them
in place for now.
This commit was SVN r10317.
Sender can receive and complete PUT request before it gets completion on the first rndv packet. senreq struct may be reused for the next MPI_Send and unexpected completion mess up the things. I sometimes got SEGV and sometimes data corruption.
This commit was SVN r10301.
btl_openib_ib_mtu and btl_mvapi_ib_mtu MCA params by showing the valid
values what what they represent (got a question about this from Cisco
testing engineers).
This commit was SVN r10277.
UD is the Unreliable Datagram transport for Infiniband, specifically OpenIB. This BTL is derived from the existing openib BTL, which is RC (Reliable Connection) based.
Still a work in progress, as there is a lot of work left to do. Specifically, performance, scalability, and flow control need to be addressed.
Currently I'm playing around with different methods for handling receive buffers, as well as profiling to figure out where the time is going.
This commit was SVN r10271.
1) don't need tree if memory is just malloc'd
2) fix memory and free list leak..
3) deregister first and then free... doh..
This commit was SVN r10251.
Added a tree to track memory allocation from MPI_Alloc_mem, this allows us to
free the registrations in a sane fashion.. also should be faster..
This commit was SVN r10248.
support for progress threads, so we shouldn't build them or try to use
them when support for progress threads has been requested. The TCP, GM,
SELF, and SM BTLs should have progress thread support, so they aren't
disabled. The Portals BTL isn't compiled on platforms with threads,
so it doens't need to be updated.
This commit was SVN r10156.
Do this rather than the my_list pointer because we need to do some
things that are somewhat special because we pre-pin eager fragments but
not send fragments. Also makes a couple ideas I have slightly easier to
play around with.
This commit was SVN r10127.
Instead of figuring out which free list the fragment belongs to based on size
we simply store a pointer to the list which it belongs in the fragment.
This was reviewed by Brian and should hit all the branches.
This commit was SVN r10072.
Trying to remember what I did here.. eager/max messages should work now, no RDMA yet. A number of other fixes and cleanups.
I do know of two problems:
Bad stuff happens when flooded with send frags too quickly - the BTL doesn't handle flow control.
Certain IBM tests turn up a length assertion in the datatype engine - needs more investigation.
This commit was SVN r10070.
2. fix for MPI_Free_mem, was calling deregister but never called mpool_free.. so
we leaked memory. Still an open issue here though, if the memory is alloc'd
and the mpool doesn't create and cache a registration, we will never find the
mpool to free with.
This commit was SVN r9944.
derefence through it. It is legal for endpoint_addr to be NULL in the
destructor because if btl_tcp_add_procs() -> btl_tcp_proc_insert()
returns UNREACH, then endpoint_addr will be NULL and we'll OBJ_RELEASE
it.
This commit was SVN r9940.
apparently don't work properly: r9869, r9868 (sm btl alignment issues)
This commit was SVN r9936.
The following SVN revision numbers were found above:
r9868 --> open-mpi/ompi@9b985c3216
r9869 --> open-mpi/ompi@adedf511fb
easier to do event accounting that way
* greatly increase receive event and buffer sizes. We're still about half
of what Cray defaults to, so I don't feel bad about the increases
* Implement a pre-pinning optimization for eager fragments - will be
pinned on first use and left pinned for the life of the fragment
* Since we can't have two receive frag callbacks fired at the same time,
don't have receive free list - just keep one receive fragment in the
module. Saves a big free list and all that interaction.
This commit was SVN r9915.
things I found:
- Locking should prevent it from happening (I think), but there was a
race condition in the component progress -- a callback could be
triggered that would free the request before it was off the outstanding
requests list.
- When pulling a request off the component free list, make sure to
reinitialize the free_called state on the IO request. This was
what was causing Edgar's failures
- In the request cleanup code, pull the request out of the per-
component free list before returning to the free list. This
probably would cause asserts to fire, although it looks like
I wrote the loops such that it would have been memory safe if
the asserts didn't fire. Not really sure why I did that, but
let's try it again...
This should go to the v1.0 and v1.1 branches.
This commit was SVN r9913.
right now. Some testing on large NUMA machines should be done in order
to make sure that we need to export this variable out to the MCA layer.
This commit was SVN r9868.
- We had a bad conditional choice, such that asking for pvfs2 would
result in pvfs trying to build as well, which was going to fail.
- We didn't try to link in the libray for PVFS2's adio component.
- We were clobbering romio_flags, so it was impossible to pass
flags to romio (like the selection of filesystems)
This commit was SVN r9854.
- LDFLAGS set at the top level of Open MPI were not passed to the
ROMIO configure script
- If ROMIO was explicitly required (with --enable-io-romio) and
not able to be built, abort OMPI's configure script.
This needs to go to the v1.0 and v1.1 branches.
This commit was SVN r9845.
- change MASK behavior for tags - we need the upper bit to be whether
the tag is reseved or not. MPI_ANY_TAG should not pull off any
reserved tag communication
- some other random debugging output to try to get some idea what is
spewing out of here.
This commit was SVN r9844.
message from a known peer (not MPI_ANY_SOURCE) then we can attach the
remote proc and initialize the convertor as soon as we know the data-type,
and the count (so basically in the _INIT macro). If it's not the case, then
create them in the _MATCHED macro (as in the original version). Of course,
beforeinitializing the convertor we check that there will be some data
in the message.
This commit, plus the convertor improvements from few days ago, lower the
latency for my test case environment (mvapi) by 0.1 microseconds. The convertor
now is as slim as it can be, I don't think there is anything else to
remove/improve.
This commit was SVN r9843.
or the bit-wise or changes all the high bits, which is bad
* push convertor creation to init to save a bit of time
* make debugging use macros so that it can go bye-bye
This commit was SVN r9810.
paper. This is just the shell, for checkpoint. Changes:
* Fix copyrights
* remove cancel code and ptl references
* add dump command
This commit was SVN r9779.
the memory descriptor is closing itself, so that it actually works properly ;). I think I
was just getting lucky and not sending enough short messages with the reference impl.
This commit was SVN r9748.
* It appears that Cray's SeaStar has some horrible performance for iovecs - IN_pLACE
was actually slower than copying into eager frags. Ugh. And we don't even pre-pin
eager frags yet!
This commit was SVN r9738.
pointer. Otherwise, we can end up segfaulting when the memory area is
used by the caller. Fixes a bug reported by Alex Spiegel.
This commit was SVN r9692.
but causes resource exhaustion on the Red Storm implementation. Sigh...
This commit was SVN r9686.
The following SVN revision numbers were found above:
r9005 --> open-mpi/ompi@20d06e889e
(1) As pointed out by Torsten after Jeff comment that there are 15 collectives yesterday.. nope.. I have 16 but
miss counted them in my ifdefs (I had two #11s). Replaces with enum...
(2) Added a readonly MCA param for how many backend algorithms are available per collective (used by benchmarker/STS)
This allowed me to remove the tuned query internal functions and replace them with ompi_coll_tuned_forced_max_algorithms[COLL].
(3) I was reading the user forced MCA params for the collectives on each comm create (module init) but I then put the
values into a global set of variables (like ompi_coll_tuned_reduce_forced_algorithm).
To fix this and make the code neater:
(a) The component looks up the MCA param indices on Open if dynamic_rules is set via the
ompi_coll_tuned_COLLECTIVE_intra_check_forced_init () call.
(b) Got rid of the ompi_coll_ompi_coll_tuned_COLLECTIVE_forced_algorithm/segmentsize/etc globals with a struct that
is now cached on the module data hung off the communicator. i.e. done right.
(c) On module init if dynamic rules enabled we call a general getvalues routine (in coll_tuned_forced.c) to get the
CURRENT values using the MCA param indices and then put them on the modules data segment.
A shorter version of getvalues exists for barrier which only needs the algorithm choice
This commit was SVN r9663.
- Some initial work on prepare_src
- Move some fragment initialization around
- Fix a union casting issue on picky compilers, identified by Don Kerr
- Other small cleanups/bugfixes
This commit was SVN r9662.
we send our local addr_t OOB. Remote side then matches endpoints and calls
dat_ep_connect(). Everything should be the same as before from here, except
that client/server roles are reversed.
- Properly set our buffer size when posting receives. When the frag used to
transfer address information is recycled by the free list, the wrong buffer
size was being used, which caused buffer overflow errors.
- Finally put the uDAPL error handling stuff in the mpool component.
- Remove a few more OPAL_OUTPUTs.
This commit was SVN r9569.
We support all the events in the PERUSE specifications, but right now only one event
of each type can be attached to a communicator. This will be worked out in the future.
The events were places in such a way, that we will be able to measure the overhead
for our threading implementation (the cost of the synchronization objects).
This commit was SVN r9500.
Not much got tested that wasn't already - I've uncovered a connection
establishment deadlock and wanted to get these changes committed before I
attack it.
The big changes:
- Moved much of the connection code from btl_udapl_component.c to
btl_udapl_endpoint.c.
- Cleaned up initialization of various fragment members.
- MCA_BTL_UDAPL_ERROR macro, which is compiled in/out appropriately.
This commit was SVN r9496.
add object size to opal class
no longer need the size when allocating a new object as this is stored in
the class structure
--- dr changes
Previous rev. maintained state on the communicator used for acking duplicate
fragments, but the communicator may be destroyed prior to successfull
delivery of an ack to the peer. We must therefore maintain this state
globally on a per peer, not a per peer, per communicator basis.
This requires that we use a global rank on the wire and translate this as
appropriate to a local rank within the communicator.
This commit was SVN r9454.
default
* Add ability to start Put and Get requests immediately instead of queuing
until synchronizaion when using Fence. Not entirely sure this is
completely safe, so it must be explicitly enabled by the user, either with
an MCA parameter or info argument to Win_create.
This commit was SVN r9418.
