This commit fixes several threading bugs:
- Add an additional lock to the btl_base_endpoint_t structure to lock
the list of pending frags. This allows the progress function to
attempt to send pending frags without needing to drop/reaquire the
lock. This should provide a small improvement in performance and
fixes a potential race between adding an removing items from the
pending list.
- Ensure fast boxes are only set up once by updating the send count
using atomics when needed and do not set the fast box buffer
pointer until the fast box is set up.
Closesopen-mpi/ompi#1408
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
* provide a more reliable way of determining that a process is a singleton by leveraging the schizo framework. Add new components for slurm, alps, and orte to detect when we are in a managed environment, and if we have been launched by mpirun or a native launcher. Set the correct envars to control ess and pmix selection in each case.
* change the relative priority of the pmix120 and pmix112 components to make pmix120 the default
* fix singleton comm-spawn by correctly setting the num_apps field of the orte_job_t created by the daemon - this fixes a segfault in register_nspace on newly created daemons
* ensure orterun doesn't propagate any ess or pmix directives in its environment
* Cleanup a few valgrind issues and memory leaks
* Fix a race condition that prevented the client from completing notification registrations (missing thread shift)
* Ensure the shizo/alps component detects launch by mpirun
This commit fixes a bug that can occur when communicating via XRC to
peers on the same node. UDCM was not saving the SRQ numbers on the
loopback endpoint (which shares its ib_addr info with all local peers)
so any messages to local peers use an invalid SRQ number.
Fixesopen-mpi/ompi#1383
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
This bug fixes two issue with the ib_addr lock:
- The ib_addr lock must always be obtained regardless of
opal_using_threads() as the CPC is run in a seperate thread.
- The ib_addr lock is held in mca_btl_openib_endpoint_connected when
calling back into the CPC start_connect on any pending
connections. This will attempt to obtain the ib_addr lock
again. Since this is not a performance-critical part of the code
the lock has been changed to be recursive.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes a bug that occurs when attempting a get or put
operation on an endpoint that is not already connected. In this case
the remote_srqn may be set to an invalid value as the rem_srqs array
on the endpoint is not populated. This commit moves the usage of the
rem_srqs array to the internal put/get functions where it is
guaranteed this array is populated.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit ensures ib_addr->remote_xrc_rcv_qp_num value is set when
creating the loopback queue pair. This is needed when communicating
with any other local peer.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes two bugs in XRC support
- When dynamic add_procs support was added to master the remote
process name was added to the non-XRC request structure. The same
value was not added to the XRC xconnect structure. This error was
not caught because the send/recv code was incorrectly using the
wrong structure member. This commmit adds the member and ensure the
xconnect code uses the correct structure.
- XRC loopback QP support has been fixed. It was 1) not setting the
correct fields on the endpoint structure, 2) calling
udcm_xrc_recv_qp_connect, and 3) was not initializing the endpoint
data.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
mca_btl_openib_put incorrectly checks the qp inline max before
allowing an inline put. This check will always fail for an endpoint
that has not been connected. The commit changes the check to use the
btl_put_local_registration_threshold instead.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Commit open-mpi/ompi@400af6c52d
introduced a regression in XRC support. The commit reversed the
ordering of shared receive queue (SRQ) and completion queue (CQ)
completion. CQ creation must always preceed SRQ creation when using
XRC as the CQs are needed to create the SRQs. This commit fixes the
ordering so that CQs are always created before SRQs.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Cleanup the configury so we properly check for Singularity under the various typical use-cases
Bring the Singularity support online. We have to turn "off" the sm BTL as it segfaults from inside the container - root cause remains unclear. Also turned "off" the various OPAL shmem components in case they are involved and someone else tries to use them. Happily, the vader BTL works just fine!
These changes fix issue https://github.com/open-mpi/ompi/issues/1336
- improve abstractions: opal/memory/linux component should be single place that opeartes with
Memory Allocation Hooks.
- avoid collisions in case dynamic component open/close: it is safe because it is linked statically.
- does not change original behaivour.
0715802f52c24c236700ac085090d5441524644c missed that there is a call
to a common/verbs_usnic symbol in the common/verbs component. This
call needs to be compiled out when the common/verbs_usnic component is
not built.
This component is a workaround to a bug in libibverbs that prints a
dire warning that usNIC devices are not supported (of course not --
usNIC devices provide functionality through libfabric, not
libibverbs). This component was written before a better workaround
was created: a "no op" libibverbs plugin for usNIC devices
(https://github.com/cisco/libusnic_verbs, and is also available in
binary form on cisco.com).
Hence, this component no longer builds by default. It's still
available if a user specifically asks for it (e.g., if they do not
want to install the "no op" libibverbs plugin), but it's not the
default. This component also has the side-effect of making
libopen-pal.so depend on libibverbs.so, which can be annoying for
packagers (which is another reason it isn't built by default any
more).
The send code in the ugni btl has an optimization that enables it to
return 1 (fragment gone) in some cases. This optimization involved
removing the btl ownership and callback flags to ensure the fragment
stuck around long enough for its completion flag to be checked. This
works fine for the single-threaded case but not in the multi-threaded
case. It is possible that a fragment will be completed by another
thread while a thread is in mca_btl_ugni_send. This competition can
lead to a leaked fragment, missed callback, or both. To fix the issue
without removing the optimization a reference count has been added to
the fragment. Callbacks and fragment release will not be made until
the fragment reference count has reach 0. The count is incremented
before sending the frag and decremented after the completion flag has
been checked. The fix has been verified to work using a multi-threaded
RMA benchmark with the osc/pt2pt component.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes a race condition that can cause an endpoint to be
added to the wait list multiple times. To fix the issue an additional
check has been added to ensure the endpoint is not on the wait list
after the wait list lock is held. The wait list processing code has
also been updated to keep the wait list lock until all wait listed
endpoints have been handled. This reduces the chance that an endpoint
that is being processed by the wait list code is not re-added to the
list by a competing send.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Three minor updates from the code review of
https://github.com/open-mpi/ompi-release/pull/933:
* Remove an extra blank line a show_help message
* We no longer allow -1 for the MCA param btl_usnic_av_eq_num, so
change the flag to REGINT_GE_ONE
* Change "num_blocks" definition to be in terms of block_len (not
eq_size)
A bunch of empirical testing has shown that increasing the retranmit
timeout from 1ms to 5ms doesn't adversely affect performance, yet
decreases the number of gratuitious retransmissions.
Add endpoints in a blocked manner so that we don't overrun the
fi_av_insert() event queue. Also make the AV EQ length an MCA param,
and report it in mca_btl_base_verbose >=5 output.
Sequence numbers will wrap around; it is not sufficient to check for
(seq-1) -- must use the SEQ_DIFF macro to properly handle the
wraparound.
This bug wasn't serious; it just meant we might retransmit one or two
extra times when retransmits were triggerd and the sequence numbers
wrapped around their sliding windows.
when SMT is enabled, a core must be counted as long as one of its hwthread is allowed
Thanks Ben Menadue for the report.
This fixes a regression from open-mpi/ompi@6d149554a7
