The 2 sided communication support is added for non-tagmatching provider
to take advantage of this BTL and PML OB1. The current state is
"functional" and not optimized for performance.
Two sided support is disabled by default and can be turned on by mca
parameter: "mca_btl_ofi_mode".
Signed-off-by: Thananon Patinyasakdikul <thananon.patinyasakdikul@intel.com>
(cherry picked from commit 080115d44069e0c461a1af105cd41f28849cdffc)
Signed-off-by: Brian Barrett <bbarrett@amazon.com>
These op codes used to be in bits/ipc.h but were removed in glibc in 2015
with a comment saying they should be defined in internal headers:
https://sourceware.org/bugzilla/show_bug.cgi?id=18560
and when glibc uses that syscall it seems to do so from its own definitions:
https://github.com/bminor/glibc/search?q=IPCOP_shmat&unscoped_q=IPCOP_shmat
So I think using #ifndef and defining them if they're not already defined
using the values from glibc is the best option.
At IBM it was the testing on redhat 8 that found this as an issue
(the opcodes being undefined on the system made the #define HAS_SHMDT
evaluate to false so intercept_shmat / intercept_shmdt were
left undefined so shmat/shmdt memory events went unintercepted).
(cherry picked from commit e8fab058dac7300569cb54b08e5500115f8bab8f)
Signed-off-by: Mark Allen <markalle@us.ibm.com>
* `libevent_core.so` contains the core functionality that we depend upon
- `libevent.so` library has been identified as the legacy target.
- `libevent_core.so` exists as far back as Libevent 2.0.5 (oldest supported by OMPI)
* `libevent_pthreads.so` can work with either `-levent` or `-levent_core`
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
* LSF ships a `libevent.so` that is no related to the `libevent.so`
shipped with Libevent.
* Add some checks to the configure logic to detect scenarios where this
conflict can be detected, and provide the user with a descriptive
warning message.
- When detected by `event/external` this is just a warning since
the internal component may be able to be used instead.
- This happens when the user supplies the LSF path via the
`LDFLAGS` envar instead of via `--with-lsf-libdir`.
- When detected by a LSF component and LSF was explicitly requested
then this becomes an error. Otherwise it will just print the warning
and that component will fail to build.
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
* This should have been `LDFLAGS` not `LIBS`. Either works, but
`LDFLAGS` is more correct. We should also include `CPPFLAGS`
just in case the header is important to the check.
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
Keep track of the connected procs in vader_add_procs().
Otherwise, the same rank will reconnect the same shmem
segment (rank 0+...) multiple times instead of the next
one as intended.
Signed-off-by: Austen Lauria <awlauria@us.ibm.com>
(cherry picked from commit f69c8d6819dcc14f471cea90b50dc8ca98de12d4)
Build was broken by mistake in commit d40662edc41a5a4d09ae690b640cfdeeb24e15a1
Fixes#7362
Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit 907ad854b46b42ae7cb1e9c87238691a5cc25e36)
Both opal_hwloc_base_get_relative_locality() and _get_locality_string()
iterate over hwloc levels to build the proc locality information.
Unfortunately, NUMA nodes are not in those normal levels anymore since 2.0.
We have to explicitly look a the special NUMA level to get that locality info.
I am factorizing the core of the iterations inside dedicated "_by_depth"
functions and calling them again for the NUMA level at the end of the loops.
Thanks to Hatem Elshazly for reporting the NUMA communicator split failure
at https://www.mail-archive.com/users@lists.open-mpi.org/msg33589.html
It looks like only the opal_hwloc_base_get_locality_string() part is needed
to fix that split, but there's no reason not to fix get_relative_locality()
as well.
Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
(cherry picked from commit ea80a20e108cb69efc67ad04ad968da7b85772af)
This PR removes the constant defining the max attachment address and
replaces it with the largest address that shows up in /proc/self/maps.
This should address issues found on AARCH64 where the max address
may differ based on the configuration.
Since the calculated max address may differ between processes the
max address is sent as part of the modex and stored in the endpoint
data.
Signed-off-by: Nathan Hjelm <hjelmn@google.com>
(cherry picked from commit 728d51f9f3f2df6577e5f9729b9d6a0fe9441d37)
This commit fixes an issue discovered in the XPMEM registration cache. It
was possible for a registration to be invalidated by multiple threads
leading to a double-free situation or re-use of an invalidated registration.
This commit fixes the issue by setting the INVALID flag on a registation
when it will be deleted. The flag is set while iterating over the tree
to take advantage of the fact that a registration can not be removed
from the VMA tree by a thread while another thread is traversing the VMA
tree.
References #6524
References #7030Closes#6534
Signed-off-by: Nathan Hjelm <hjelmn@google.com>
(cherry picked from commit f86f805be1145ace46b570c5c518555b38e58cee)
Clarify in README what --with-hwloc does in its different use cases.
Also, ensure that the behavior when specifying `--with-hwloc` is the
same as if that option is not specified at all. This is what we did
in Open MPI <= v3.x; looks like we inadvertantly caused `--with-hwloc`
to be synonymous with `--with-hwloc=external` in v4.0.0.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit 18c3e1af5ef281e8b502a7ad778256889cc707d1)
syscall() returns a long, but we are invoking shmat(), which returns
a void*.
Signed-off-by: Maxwell Coil <mcoil@nd.edu>
(cherry picked from commit 52a9cce6f3dcd87e9ae66177398b60b9317e9339)
This commit fixes a configure bug that caused flow control to be
disabled regardless of the configure options used.
Signed-off-by: Todd Kordenbrock <thkgcode@gmail.com>
(cherry picked from commit f7e74b6a3d19cd6e3edc3364783311e595f81b3d)
The prior implementation was simply wrong.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit 53ebea12aa98ba5440924a8ee914f6707f63608b)
related to #7128
The UCX crew is no longer guaranteeing that the UCT API is going to be frozen,
so this is kind of a whack-a-mole problem trying to keep the BTL UCT working
with various changing UCT APIs.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
(cherry picked from commit 9d345d9aa000233bec148540b071cecffc94438c)
OpenUCX broke the UCT API again in v1.8. This commit updates
btl/uct to fix compilation with current OpenUCX master
(future v1.8). Further changes will likely be needed for
the final release.
Signed-off-by: Nathan Hjelm <hjelmn@google.com>
(cherry picked from commit 526775dfd7ad75c308532784de4fb3ffed25458f)
It was previously accidentally set to 0.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit 132e4cab3bc71df0da87368a332d6af0090a6977)
Move the prefix area from the head to the body in relevant size
computations. This fixes a problem in high traffic situations where
usNIC may have sent from unregistered memory.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit fe7f772f21627b01838c007db7cedbbb0ce8b536)
New MCA param: btl_usnic_max_resends_per_iteration. This is the max
number of resends we'll do in a single pass through usNIC component
progress. This prevents progress from getting stuck in an endless
loop of retransmissions (i.e., if more retransmissions are triggered
during the sending of retransmissions). Specifically: we need to
leave the resend loop to allow receives to happen (which may ACK
messages we have sent previously, and therefore cause pending resends
to be moot).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit 27e3040dfeba00a9a2615a217c164899f0009e59)
Significantly increase the default retrans timeout. If the
retrans timeout is too soon, we can end up in a retransmission storm
where the logic will continually re-transmit the same frames during a
single run through the usNIC progress function (because the timer for
a single frame expires before we have run through re-transmitting all
the frames pending re-transmission).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit 3cc95d86b2123f38f392e56adca7ac8a1fef6454)
New MCA parameter: btl_usnic_ack_iteration_delay. Set this to the
number of times through the usNIC component progress function before
sending a standalone ACK (vs. piggy-backing the ACK on any other send
going to the target peer).
Use "ticks" language to clarify that we're really counting the number
of times through the usNIC component DATA_CHANNEL completion check (to
check for incoming messages) -- it has no relation to wall clock time
whatsoever.
Also slightly change the channel-checking scheme in usNIC component
progress: only check the PRIORITY channel once (vs. checking it once,
not finding anything, and then falling through the progress_2() where we
check PRIORITY again and then check the DATA channel).
As before, if our "progress" libevent fires, increment the tick
counter enough to guarantee that all endpoints that need an ACK will
get triggered to send standalone ACKs the next time through progress,
if necessary.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit 968b1a51b59898877a8c7268d463d3d7d78d86a3)
Rename "get_nsec()" to "get_ticks()" to more accurately reflect that
this function has no correlation to wall clock time at all.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
(cherry picked from commit ce2910a28aea61043b81324c67999f3a47cfe7ac)
Trying out to run processes via mpirun in Podman containers has shown
that the CMA btl_vader_single_copy_mechanism does not work when user
namespaces are involved.
Creating containers with Podman requires at least user namespaces to be
able to do unprivileged mounts in a container
Even if running the container with user namespace user ID mappings which
result in the same user ID on the inside and outside of all involved
containers, the check in the kernel to allow ptrace (and thus
process_vm_{read,write}v()), fails if the same IDs are not in the same
user namespace.
One workaround is to specify '--mca btl_vader_single_copy_mechanism none'
and this commit adds code to automatically skip CMA if user namespaces
are detected and fall back to MCA_BTL_VADER_EMUL.
Signed-off-by: Adrian Reber <areber@redhat.com>
(cherry picked from commit fc68d8a90fe86284e9dc730f878b55c0412f01d2)
This commit changes how the single-copy emulation in the vader btl
operates. Before this change the BTL set its put and get limits
based on the max send size. After this change the limits are unset
and the put or get operation is fragmented internally.
References #6568
Signed-off-by: Nathan Hjelm <hjelmn@google.com>
(cherry picked from commit ae91b11de2314ab11a9842d9738cd14f8f1e393b)
Commit d7053a3 broke things for the case when Open MPI 4.0.x is built
without UCX support. Problem was it was trying to partially initialize
the btl to try and delay printing of a help message till wireup. Well
this sort of doesn't work in all cases. Rather than keep piling on
changes to support a help message for a BTL that we are deprecating, take
a keep it simple stupid approach.
So, revert most of d7053a3 and instead put the help message back in the
original location, during scan of ports of the available HCAs to check
for whether or not link layer for that port is configured for ethernet or infiniband.
If Open MPI was built with UCX support, don't emit the help message, if
UCX was not linked in, emit the help message.
Verified on a system with connectX5 HCAs configured with two ports configured
for ethernet and two for infiniband.
relates to #6785
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
