The send inline optimization uses the btl_sendi function to achieve lower
latency and higher message rates. Before this commit BTLs were allowed to
assume the descriptor was non-NULL and were expected to return a valid
descriptor if the send could not be completed using btl_sendi. This
behavior was fine until the usage of btl_sendi was changed in ob1. This
commit allows the caller to specify NULL for the descriptor. The affected
btls have been updated to handle this case.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds an interface for btl's to export support for 64-bit atomic
operations on integers. BTL's that can support atomic operations should
implement these functions and set the appropriate btl_flags and btl_atomic_flags.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The old BTL interface provided support for RDMA through the use of
the btl_prepare_src and btl_prepare_dst functions. These functions were
expected to prepare as much of the user buffer as possible for the RDMA
operation and return a descriptor. The descriptor contained segment
information on the prepared region. The btl user could then pass the
RDMA segment information to a remote peer. Once the peer received that
information it then packed it into a similar descriptor on the other
side that could then be passed into a single btl_put or btl_get
operation.
Changes:
- Added functions to register and deregister memory regions with the
btl. If no registration is needed a btl should set these function
pointers to NULL. These function take over for btl_prepare_src/dst
and btl_free for RDMA operations. The caller should specify the
maximum permissions needed on the memory.
- Changed the function signatures for both btl_put and btl_get. In
place of a prepared descriptor the caller should provide the source
and destination addresses and registration handles as well as a
new callback function. The callback will be provided with the local
address and registration handle, callback context, callback data, and
status. See mca_btl_base_rdma_completion_fn_t in btl.h.
- Added a new btl constraint: MCA_BTL_REG_HANDLE_MAX_SIZE. This
value specifies the maximum size of any btl's registration handle.
- Removed the btl_prepare_dst function. This reflects the fact that
RDMA operations no longer depend on "prepared" descriptors.
- Removed the btl_seg_size member. There is no need to btl's to
subclass the mca_btl_base_segment_t class anymore.
- Expose the btl's put/get limitations with new struct members:
btl_put_limit, btl_put_alignment, btl_get_limit, btl_get_alignment.
- Remove the mca_mpool_base_registration_t argument from the btl_prepare_src
function. The argument was intended to support RDMA operations and is no
longer necessary.
- Remove des_remote/des_remote_count from the mca_btl_base_descriptor_t
structure. This structure member was originally used to specify the remote
segment for RDMA operations. Since the new btl interface no longer uses
desriptors for RDMA this member no longer has a purpose. In addition
to removing these members the local segment structure fields have been
renamed to from des_local/des_local_count to des_segments/des_segment_count.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Per feedback from rhc, manually set the base_ptr member
of the opal_buffer_t variable to NULL prior to calling
OBJ_RELEASE. A similar feature of opal_dss.load also
exists so likewise reset the base_ptr to NULL prior to
invoking it.
Hopefully the opal_buffer_t struct does not change
frequently.
Minor cleanups to reduce output when pmix_base_verbose
mca paramater is set.
usnic_fls() can actually return 0, leading us to incorrectly free() a
buffer instead of OMPI_FREE_LIST_RETURN_MT'ing it.
So add an explicit bool in the struct that tracks whether the buffer
came from malloc or a freelist.
This was CID 1269660.
Coverity alerted us to the fact that there are places where
the synonym_for param is hard-coded to -1 when calling
register_variable(). It would be a coding error if synonym_for==-1
and (flags & MCA_BASE_VAR_FLAG_SYNONYM)>0, so let's add that to the
debug-only check at the top of the function.
This was CID 993717.
Remove use of the Cray PMI KVS - which is designed for a lighweight
MPI that exchanges only a minimimal amount of connection info
(about 128 bytes per rank) - within cray/pmix. Use Cray PMI
collective extensions instead.
This is the first of several steps to accelerate launch of
Open MPI on Cray systems using either native aprun or nativized
slurm.
in this code and it ended up changing the logic that is used to set up eager RDMA.
Rather than setting up eager RDMA with a high priority message, it did it the other
way around. For some reason, CUDA-aware support did not like this. So, basically,
restore the logic to the way it was prior to the refactoring. The refactoring did not
intend to change this. Lightly reviewed by hjelmn.
