The send inline optimization uses the btl_sendi function to achieve lower
latency and higher message rates. Before this commit BTLs were allowed to
assume the descriptor was non-NULL and were expected to return a valid
descriptor if the send could not be completed using btl_sendi. This
behavior was fine until the usage of btl_sendi was changed in ob1. This
commit allows the caller to specify NULL for the descriptor. The affected
btls have been updated to handle this case.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds an interface for btl's to export support for 64-bit atomic
operations on integers. BTL's that can support atomic operations should
implement these functions and set the appropriate btl_flags and btl_atomic_flags.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The old BTL interface provided support for RDMA through the use of
the btl_prepare_src and btl_prepare_dst functions. These functions were
expected to prepare as much of the user buffer as possible for the RDMA
operation and return a descriptor. The descriptor contained segment
information on the prepared region. The btl user could then pass the
RDMA segment information to a remote peer. Once the peer received that
information it then packed it into a similar descriptor on the other
side that could then be passed into a single btl_put or btl_get
operation.
Changes:
- Added functions to register and deregister memory regions with the
btl. If no registration is needed a btl should set these function
pointers to NULL. These function take over for btl_prepare_src/dst
and btl_free for RDMA operations. The caller should specify the
maximum permissions needed on the memory.
- Changed the function signatures for both btl_put and btl_get. In
place of a prepared descriptor the caller should provide the source
and destination addresses and registration handles as well as a
new callback function. The callback will be provided with the local
address and registration handle, callback context, callback data, and
status. See mca_btl_base_rdma_completion_fn_t in btl.h.
- Added a new btl constraint: MCA_BTL_REG_HANDLE_MAX_SIZE. This
value specifies the maximum size of any btl's registration handle.
- Removed the btl_prepare_dst function. This reflects the fact that
RDMA operations no longer depend on "prepared" descriptors.
- Removed the btl_seg_size member. There is no need to btl's to
subclass the mca_btl_base_segment_t class anymore.
- Expose the btl's put/get limitations with new struct members:
btl_put_limit, btl_put_alignment, btl_get_limit, btl_get_alignment.
- Remove the mca_mpool_base_registration_t argument from the btl_prepare_src
function. The argument was intended to support RDMA operations and is no
longer necessary.
- Remove des_remote/des_remote_count from the mca_btl_base_descriptor_t
structure. This structure member was originally used to specify the remote
segment for RDMA operations. Since the new btl interface no longer uses
desriptors for RDMA this member no longer has a purpose. In addition
to removing these members the local segment structure fields have been
renamed to from des_local/des_local_count to des_segments/des_segment_count.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
usnic_fls() can actually return 0, leading us to incorrectly free() a
buffer instead of OMPI_FREE_LIST_RETURN_MT'ing it.
So add an explicit bool in the struct that tracks whether the buffer
came from malloc or a freelist.
This was CID 1269660.
in this code and it ended up changing the logic that is used to set up eager RDMA.
Rather than setting up eager RDMA with a high priority message, it did it the other
way around. For some reason, CUDA-aware support did not like this. So, basically,
restore the logic to the way it was prior to the refactoring. The refactoring did not
intend to change this. Lightly reviewed by hjelmn.
This commit adds an MCA variable to select Portals4 logical
addressing, populates the logical-to-physical mapping table and
initializes the NI in this mode.
Ensure to count *this* process when checking for how many VFs we need
on the local server.
(cherry picked from commit 386c01934e98cb8dcb48ff648ecdfb0c8677baa9)
If there are not enough resources (e.g., low VFs), we can end up
calling finalize_one_channel() on the same channel multiple times. So
ensure to NULL out fields that we have freed already so that we do not
try to free them a second time.
Fixes CSCus26648.
Fix the ordering so that we obtain the usnic netmask information
*before* we do the filtering based on CIDR-specified networks.
Also requires upstream Github libfabric commit 3976745.
Fixes CSCus22495.
