interconnects that provide matching logic in the library.
Currently includes support for MX and some support for
Portals
* Fix overuse of proc_pml pointer on the ompi_proc structuer,
splitting into proc_pml for pml data and proc_bml for
the BML endpoint data
* bug fixes in bsend init code, which wasn't being used by
the OB1 or DR PMLs...
This commit was SVN r10642.
mpi_leave_pinned when multiple OpenIB HCA ports are found.
Specifically, if mpi_leave_pinned == 1 and ultiple HCA ports are
found, the MCA parameter btl_openib_max_btls is set to 1. If the MCA
parameter btl_openib_warn_leave_pinned_multi_port is true, emit a
warning that this happened (having an MCA parameter to control the
warning allows users/sysadmins to turn it off instead of being nagged
for every run).
This commit was SVN r10521.
time.
UD is connectionless, and as long as peers are statically assigned to QPs,
there is no reason to set up the adressing information lazily.
Lots of code was axed, as endpoints no longer have state. Removed a
number of other elements in the endpoint struct to make it as lightweight
as possible.
I was able to remove an entire function call/branch in the send path,
which I believe is the main contributor to a 2us drop in NetPIPE latency.
Some whitespace cleanups as well.
Passes IBM test suite, and all but certain intel tests that were failing
before the change, over ob1 PML.
This commit was SVN r10494.
Moved a lot of the module-specific init from the component init to the module init.
Try keeping a pointer to reduce indexing, didn't seem to help - leaving in place
for now.
This commit was SVN r10485.
Playing around with OPAL_LIKELY/UNLIKELY, no real gains yet.
Reworked progress() to process many WC's at a time, as well
as immediately repost groups of receive buffers.
This commit was SVN r10481.
1. ompi/mca/btl/udapl/btl_udapl_proc.c should be including
btl_udapl_endpoint.h for mca_btl_udapl_proc_insert function.
2. btl_udapl_endpoint.c it looks like you are using
&endpoint->endpoint_lock when you should use &ep->endpoint_lock in a
OPAL_THREAD_LOCK call.
3. btl_udapl_frag.h has a couple opal_list_item_t's that should be
ompi_free_list_item_t in the _FRAG_ALLOC_{EAGER,MAX} macros.
This commit was SVN r10442.
mpi_leave_pinned when multiple OpenIB HCA ports are found.
Specifically, if mpi_leave_pinned == 1 and ultiple HCA ports are
found, the MCA parameter btl_openib_max_btls is set to 1. If the MCA
parameter btl_openib_warn_leave_pinned_multi_port is true, emit a
warning that this happened (having an MCA parameter to control the
warning allows users/sysadmins to turn it off instead of being nagged
for every run).
This commit was SVN r10424.
send/receive queues if there is something available in the short message
rdma queues, then we have to poll *ALL* the rdma queues before exiting,
or we aren't fair about frag reception and fall into degenerate matching
cases.
This commit was SVN r10410.
- Added some basic flow control to limit number of posted sends.
- Merged endpoint send/recv lock into single endpoint lock.
- Set the LMR triplet length in the send path, not at allocation time.
This has to be done because upper layers might send less than the
amount allocated.
- Alter the tie-breaker if statement protecting the second call
to dat_ep_connect(). The logic was reversed compared to the tie-
breaker for the first dat_ep_connect(), making it possible for
3 or more processes to form a deadlock loop.
- Some asserts were added for debugging purposes.. leaving them
in place for now.
This commit was SVN r10317.
btl_openib_ib_mtu and btl_mvapi_ib_mtu MCA params by showing the valid
values what what they represent (got a question about this from Cisco
testing engineers).
This commit was SVN r10277.
UD is the Unreliable Datagram transport for Infiniband, specifically OpenIB. This BTL is derived from the existing openib BTL, which is RC (Reliable Connection) based.
Still a work in progress, as there is a lot of work left to do. Specifically, performance, scalability, and flow control need to be addressed.
Currently I'm playing around with different methods for handling receive buffers, as well as profiling to figure out where the time is going.
This commit was SVN r10271.
support for progress threads, so we shouldn't build them or try to use
them when support for progress threads has been requested. The TCP, GM,
SELF, and SM BTLs should have progress thread support, so they aren't
disabled. The Portals BTL isn't compiled on platforms with threads,
so it doens't need to be updated.
This commit was SVN r10156.
Do this rather than the my_list pointer because we need to do some
things that are somewhat special because we pre-pin eager fragments but
not send fragments. Also makes a couple ideas I have slightly easier to
play around with.
This commit was SVN r10127.
Instead of figuring out which free list the fragment belongs to based on size
we simply store a pointer to the list which it belongs in the fragment.
This was reviewed by Brian and should hit all the branches.
This commit was SVN r10072.
Trying to remember what I did here.. eager/max messages should work now, no RDMA yet. A number of other fixes and cleanups.
I do know of two problems:
Bad stuff happens when flooded with send frags too quickly - the BTL doesn't handle flow control.
Certain IBM tests turn up a length assertion in the datatype engine - needs more investigation.
This commit was SVN r10070.
derefence through it. It is legal for endpoint_addr to be NULL in the
destructor because if btl_tcp_add_procs() -> btl_tcp_proc_insert()
returns UNREACH, then endpoint_addr will be NULL and we'll OBJ_RELEASE
it.
This commit was SVN r9940.
apparently don't work properly: r9869, r9868 (sm btl alignment issues)
This commit was SVN r9936.
The following SVN revision numbers were found above:
r9868 --> open-mpi/ompi@9b985c3216
r9869 --> open-mpi/ompi@adedf511fb
easier to do event accounting that way
* greatly increase receive event and buffer sizes. We're still about half
of what Cray defaults to, so I don't feel bad about the increases
* Implement a pre-pinning optimization for eager fragments - will be
pinned on first use and left pinned for the life of the fragment
* Since we can't have two receive frag callbacks fired at the same time,
don't have receive free list - just keep one receive fragment in the
module. Saves a big free list and all that interaction.
This commit was SVN r9915.
right now. Some testing on large NUMA machines should be done in order
to make sure that we need to export this variable out to the MCA layer.
This commit was SVN r9868.
the memory descriptor is closing itself, so that it actually works properly ;). I think I
was just getting lucky and not sending enough short messages with the reference impl.
This commit was SVN r9748.
* It appears that Cray's SeaStar has some horrible performance for iovecs - IN_pLACE
was actually slower than copying into eager frags. Ugh. And we don't even pre-pin
eager frags yet!
This commit was SVN r9738.
but causes resource exhaustion on the Red Storm implementation. Sigh...
This commit was SVN r9686.
The following SVN revision numbers were found above:
r9005 --> open-mpi/ompi@20d06e889e
- Some initial work on prepare_src
- Move some fragment initialization around
- Fix a union casting issue on picky compilers, identified by Don Kerr
- Other small cleanups/bugfixes
This commit was SVN r9662.
we send our local addr_t OOB. Remote side then matches endpoints and calls
dat_ep_connect(). Everything should be the same as before from here, except
that client/server roles are reversed.
- Properly set our buffer size when posting receives. When the frag used to
transfer address information is recycled by the free list, the wrong buffer
size was being used, which caused buffer overflow errors.
- Finally put the uDAPL error handling stuff in the mpool component.
- Remove a few more OPAL_OUTPUTs.
This commit was SVN r9569.
Not much got tested that wasn't already - I've uncovered a connection
establishment deadlock and wanted to get these changes committed before I
attack it.
The big changes:
- Moved much of the connection code from btl_udapl_component.c to
btl_udapl_endpoint.c.
- Cleaned up initialization of various fragment members.
- MCA_BTL_UDAPL_ERROR macro, which is compiled in/out appropriately.
This commit was SVN r9496.
- Grab the mpool_registration in _frag_common_constructor()
- Save the LMR context in the segment key
- No need for cookie variables - can just cast the frag
- No need to memcpy() data when recv'ing
- Add an LMR triplet to the fragment structure and initialize it
in btl_udapl_alloc().
- Whitespace/typo fixes, remove some opal_output() calls
Looks like I can use triplets describing sub-regions of registered LMR's. So I
do this - prior to this patch I was sending the entire free list memory over,
which isn't correct :)
Back to an earlier problem - when sending address information right after
connection establishment, the receiving end receives a DTO completion event and
appears to have good data. But the sending end never receives a DTO completion
event indicating the send completed, and never completes the client side of the
connection.
This commit was SVN r9386.
In short, I'm very close to having connection establishment and eager send/recv working.
Part of the connection process involves sending address information from the
client to server. For some reason, I am never receiving an event indicating
completetion of the send on the client side. Otherwise, connection
establishment is working and eager send/recv should be trivial from here.
Some more detailed changes:
- Send partially implemented, just handles starting up new connections.
- Several support functions implemented for establishing connection. Client
side code went in btl_udapl_endpoint.c, server side in btl_udapl_component.c
- Frags list and send/recv locks added to the endpoint structure.
- BTL sets up a public service point, which listens for new connections.
Steps over ports that are already bound, iterating through a range of ports.
- Remove any traces of recv frags, don't think I need them after all.
- Pieces of component_progress() implemented for connection establishment.
- Frags have two new types for connection establishment - CONN_SEND and
CONN_RECV.
- Many other minor cleanups not affecting functionality
This commit was SVN r9345.
flag, new flags to be included when convertor is initialized
- modified pml/btl module defs and added stub functions for diagnostic
output routines to dump state of queues / endpoints
- updates to data reliability pml
This commit was SVN r9329.
- initial support for gm progress thread
- corrected threading issue in pml
- added polling progress for a configurable number of cycles to wait for threaded case
This commit was SVN r9188.
- moved hton64 and ntoh64 from the bunch of places it had been copied
into one header file
- properly set and use the btl_tcp's nbo option to put things in
network byte order on the wire if both sides don't have the same
endianness
- Put the OB1 PML's headers (with a couple exceptions I need to discuss
with Tim) in network byte order on the wire if both sides don't have
the same endianness
- since it was needed for the TCP BTL, move the orte_process_name_t
HTON and NTOH macros from the TCP OOB to ns_types.h
This commit was SVN r9145.
when running an MPI job spanning a node that has two TCP NICs and a
node that has one TCP NIC. Previously, for the 2 NIC/module process,
we would return the first peer IP address if we couldn't find a subnet
match with any of the peer's published IP addresses -- this was to
support running OMPI across subnet boundaries. Changed the behavior
to only do that behavior if the IP address we're trying to match is
public (i.e., not 10.x.y.z, 192.168.x.y, or 172.16.x.y) *and* any of
the remote peer's addresses are public (working on the assumption that
if we both have public addresses, they're routable to each other).
This definitely will not work in all scenarios, such as when we go to
WAN kinds of executions, and will need to be revisited at that time.
This commit was SVN r9119.
run-time support, as it appears to be doing bad things to memory. Update
the hack to get the local nid to match the recent TCP nal changes, and
update the P3RT api useage
This commit was SVN r9005.
- move files out of toplevel include/ and etc/, moving it into the
sub-projects
- rather than including config headers with <project>/include,
have them as <project>
- require all headers to be included with a project prefix, with
the exception of the config headers ({opal,orte,ompi}_config.h
mpi.h, and mpif.h)
This commit was SVN r8985.
- Start filling in the progress function, focusing on connection establishment.
- Initialize udapl mpool and free lists
- Create/destroy a protection zone with each IA
- Misc organization as I learn how things work
This commit was SVN r8969.
complete, but stable enough that it will have no impact on general development,
so into the trunk it goes. Changes in this commit include:
- Remove the --with option for disabling MPI-2 onesided support. It
complicated code, and has no real reason for existing
- add a framework osc (OneSided Communication) for encapsulating
all the MPI-2 onesided functionality
- Modify the MPI interface functions for the MPI-2 onesided chapter
to properly call the underlying framework and do the required
error checking
- Created an osc component pt2pt, which is layered over the BML/BTL
for communication (although it also uses the PML for long message
transfers). Currently, all support functions, all communication
functions (Put, Get, Accumulate), and the Fence synchronization
function are implemented. The PWSC active synchronization
functions and Lock/Unlock passive synchronization functions are
still not implemented
This commit was SVN r8836.
It's still not very clean as we still lock the mvapi_btl mutex inside a critical section
protected by the endpoint mutex ...
This commit was SVN r8810.
- Remove printing of CFLAGS in configure.m4
- Set MCA_BTL_FLAGS_SEND flag
- Improved error handling during module initialization
- Extract the address of each interface with dat_ia_query
- Start playing around with fragment stuff - probably wrong
- Misc code cleanup (removal of GM-specific code)
This commit was SVN r8801.
- Move mca_btl_udapl_error/mca_btl_module_init to mca_btl_udapl.c and rename it
- White space cleanups
- Free the uDAPL evd and ia handles in mca_btl_udapl_finalize
This commit was SVN r8705.
- Borrow configure.m4 from the mvapi btl. One of the uDAPL headers emits a
warning when -pedantic is enabled, so strip it out.
- Change function check in ompi_check_dapl.m4 from dat_ia_open to
dat_registry_list_providers.. dat_ia_open wasn't working right
- Make the references to prepare_dst, put, and get NULL for now
- Add opal_output() calls in all the udapl interface functions for debugging
- Add evd_qlen component parameter to control event dispatcher queue length
- First stab at component_init and module_init
- Misc cleanups - whitespace, dead code removal
- Update copyrights to 2006
This commit was SVN r8701.
On some systems (like windows) qsort can modify the order of modules when the
comparaison function return 0. As we expect to have the
mca_btl_sm_add_procs_same_base_addr called before mca_btl_sm_add_procs we have
force the qsort to return the SM modules in the correct order. Giving to the
same_addr module a slightly higher priority solve this problem.
This commit was SVN r8620.
- Copied the template BTL and renamed everything
- Compiles and shows up correctly in ompi_info, not tested past that
- Should be ignored for everyone but me
This commit was SVN r8544.
empty for non threaded builds, but somehow just by moving the code a
little bit around and removing 2 call to lock/unlock the latency for TCP
went down by 2 micro-seconds ...
This commit was SVN r8426.
number of recv system call by caching the data. Each endpoint has a buffer
(the size is an MCA parameter) that can be use as a cache. Before each receive
operation this buffer is added at the end of the iovec list. All data that are
not expected by the fragment will go in this cache. If the cache contain data
all subsequent receive will just memcpy the data into the BTL buffers.
The only drawback is that we will spin around the receive_handle until all the
cached data is readed by the PML layer. This limitation come from the fact that
the event library is unable to call us if there is no events on the socket.
Therefore we are unable to keep the data in the cache until the next loop
into the progress engine.
This commit was SVN r8398.
increase the previous connection code was broken. It can take as much as 60 seconds to connect
64 processes. Now we do not create the connections when we add the procs but only when we send
them the first message. Now it take only 1.6 seconds to setup a 64 procs MPI job over MX (doing a 2 steps barrier in order to insure that we create all the connections).
This commit was SVN r8252.
* turns out (duh!) that there was a reason that the <projectdir>dir
variable was set in the AM conditional. If not, stupid directories
are created and not needed... duh.
This commit was SVN r8205.
component/base Makefile.am files, reducing the time configure spends
stamping out Makefiles at the end
* Install base_impl.h file when devel-headers are being installed
This commit was SVN r8200.
is labeled as internal so the users will not see it but it is not read-only so we can still
play with it (that's for our internal tests). This is supposed to dissapear later after the
next (or next next) release of the MX library, but we need it now as a quick fix before the
release.
This commit was SVN r8161.
- Add big comment about a general overview of what the sm btl is doing
- random small code cleanups
- fix instances of mca_btl_sm[0] to mca_btl_sm[1] where relevant
- remove a lot of unused, confusing, and incorrect interface functions
from ompi_fifo.h and ompi_circular_buffer.h. These functions, if
they were used, would not work properly with the scheme that the sm
btl uses with the fifos (i.e., receiver makes right -- if necessary)
- add some missing offset computations in the fifo and circular buffers
- change the types of offsets to be ssize_t, not size_t
- remove an offset parameter from a function that didn't need it
This commit was SVN r8135.
- Ensure to convert base_shared_mem_flags to be a relative offset in
the global storage, and then to convert that back to an absolute
virtual address before we try to use it
- Don't double increment n_local_procs when calculating the peer rank
during bootstrapping of the different base address case
Something else is still wrong; if mmap() returns a different base
address, things don't work (i.e., segv or hang forever when you try to
send a message). More specifically, the bootstrapping now seems to
correctly handle the case when mmap() base addresses are different,
but the message passing does *not* -- it always assumes that the
mmap() base addresses are the same.
Still working on the fix for that -- want to checkpoint what has been
done so far to facilitate working on different machines...
This commit was SVN r8134.
- first we setup the connections in the begining with all the peers
- MX does not handle well the case where several peers make connections to the same
destination simultaneously.
So I change the order in which we connect. First we compute our rank in the array,
then in a round-robin fashion we setup connection starting with our left neighboard.
This commit was SVN r8075.
larger than 32K for inter-nodes transfert ... and then they do not support iovecs larger than
16K for inter-node transfert. Therefore we have to set the size of our first fragment to
16K to match both cases.
This commit was SVN r7926.
to be supported by mellanox vapi.. perhaps this will be supported in the near
future, for now it doesn't hurt to have it in the trunk
Also cleanup the receive descriptor posting macro's..
This commit was SVN r7903.
Note, we are failing the ring tests in the intel p2p test suite, but we seem
to fail the same tests under the current trunk.. will look into this further.
This commit was SVN r7823.
memory initialization, call opal_progress() to push any pending events
around and possibly yield the processor if nothing entertaining is happening.
This should probably go to the 1.0 branch.
This commit was SVN r7808.
originally suggested by Ralf Wildenhues, to try to speed autogen, configure,
and make (and possibly even make install). Use automake's include directive
to drastically reduce the number of Makefile files (although the number of
Makefile.am files is the same - most are just included in a top-level
Makefile.am). Also use an Automake SUBDIRs feature to eliminate the
dynamic-mca tree, which was no longer really needed. This makes adding
a framework easier (since you don't have to remember the dynamic-mca
tree) and makes building faster (as make doesn't have to recurse through
the dynamic-mca tree)
This commit was SVN r7777.
on others as an char*. Thus the right side of all assignment should be explicitly casted to an void* in
order to avoid any casting complaints from the compilers.
This commit was SVN r7607.
On thor, if IB is down, we get NULL back from ibv_get_devices(), which then
caused segfaults in dlist_start().
* Pretty-print error message if no HCAs found
This commit was SVN r7557.
any NICs to use
* Make mvapi, gm, and mx components all publish information, even if there
are no NICs available so that modex_recv doesn't hang. If there are no
NICs available, don't set the reachable bit, but don't do anything
to fail. This unfortunately doesn't cover the hangs that will result if
different procs load different sets of components, but it's a start
This commit was SVN r7550.
Try to detect which form this version of Open IB uses, defaulting to the 5
version if we can't figure it out (the new version has 5 params)
* Only add -lcm if it exists on the system - some versions of Open IB
apparently don't need it.
This commit was SVN r7542.
searching the tree. Modified the memory callback to search the tree at each
page boundary for registrations. This is necessary as an application may
malloc memory and send out of any portion of that memory, even discontiguous
regions.
This commit was SVN r7510.
add misc asserts to check for proper reference counting.
ugly hack 1 -- use mallopt to never release memory ala sbrk - this is
commented out in mca_btl_mvapi_component_init
ugly hack 2 -- test registrations comming out of the tree via rcache_find, for
an unknown reason the tree is returning registrations where the address is not
within the base or bound of the registration. If this happens, we return
NULL.
comment out code to enable mem hooks if leave_pinned is set, note we can do
this via an mca param and will default it to leave_pinned with mem_hooks when
we iron out these issues.
I am adding a unit test for the rcache. Note that we have a unit test for the
rb tree but the compare function is significantly different than that used for
registrations. After we have tracked down the issues with rcache_rb we will
remove the above hacks.
This commit was SVN r7499.
Red Storm is complaining about invalid memory pointer (need to go back
to Linux and look at this with valgrind)
* Turn off send in place for now, so I can run the tests on RS and see if
everything else is ok
This commit was SVN r7497.
* fix dumb bug in btl_portals_get where I using the dest descriptor key instead
of the source descriptor key for the match bits, resulting in a PtlGet() with
the wrong match bits
This commit was SVN r7496.
