Fix two facepalms:
1. The "uint32" in the hash map functions refer to the *key* size, not
the *value* size. The values are always 64 bits.
2. Pass the straight value to the "set" functions -- not the pointer
to the value.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
This commit fixes a typo where a bcast is used instead of the intended
collective (barrier).
References #5262
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit changed the way btl/ofi call progress. Before, we force
progression with every rdma/atomic call. This gives performance boost in
some case and slow down on others. Now we only force progression after
some number of rdma calls which result in better performance overall.
Also added new MCA parameter 'mca_btl_ofi_progress_threshold' to set
the threshold number. The new default is 64.
Also:
Added FI_DELIVERY_COMPLETE to tx_rtx flags to ensure that the completion
is generated after the message has been received on the remote side.
Signed-off-by: Thananon Patinyasakdikul <thananon.patinyasakdikul@intel.com>
This commit improves the injection rate and latency for RDMA
operations. This is done by the following improvements:
- If C11's _Thread_local keyword is available then always use the
same virtual device index for the same thread when using RDMA. If
the keyword is not available then attempt to use any device that
isn't already in use. The binding support is enabled by default but
can be disabled via the btl_ugni_bind_devices MCA variable.
- When posting FMA and RDMA operations always attempt to reap
completions after posting the operation. This allows us to
better balance the work of reaping completions across all
application threads.
- Limit the total number of outstanding BTE transactions. This
fixes a performance bug when using many threads.
- Split out RDMA and local SMSG completion queue sizes. The RDMA
queue size is better tuned for performance with RMA-MT.
- Split out put and get FMA limits. The old btl_ugni_fma_limit MCA
variable is deprecated. The new variable names are:
btl_ugni_fma_put_limit and btl_ugni_fma_get_limit.
- Change how post descriptors are handled. They are no longer
allocated seperately from the RDMA endpoints.
- Some cleanup to move error code out of the critical path.
- Disable the FMA sharing flag on the CDM when we detect that there
should be enough FMA descriptors for the number of virtual devices
we plan will create. If the user sets this flag we will not unset
it. This change should improve the small-message RMA performance by
~ 10%.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
so latest ROM-IO can be used with Open MPI.
Note this first and naive implementation does not use the wait_fn callback.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
This commit adds a new btl for one-sided and two-sided. This btl
uses the uct layer in OpenUCX. This btl makes use of multiple uct
contexts and per-thread device pinning to provide good performance
when using threads and osc/rdma. This btl has been tested extensively
with osc/rdma and passes all MTT tests on aries and IB hardware.
For now this new component disables itself but can be enabled by
setting the btl_ucx_transports MCA variable with a comma-delimited
list of supported memory domains/transport layers. For example:
--mca btl_uct_memory_domains ib/mlx5_0. The specific transports used
can be selected using --mca btl_uct_transports. The default is to use
any available transport.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The osc/rdma module did not wait for all pending atomics to complete
before tearing down. This could lead to weird issues as the target
location may no longer be registered or allocated.
This commit also fixes an offset calculation issue in
ompi_osc_get_data_blocking ().
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The giant size of the TCP proc struct is causing a problem in some
environments (because it is allocated on the stack), and it was too
big, anyway.
Instead, use a hash map. That way, it starts small and can grow if it
needs to. It also makes no assumptions about the values of the kernel
interface indexes.
Fixes#5292.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
a bug sneaked into constructing the list of aggregators
processes when using the fileview based grouping options
Signed-off-by: Edgar Gabriel <egabriel@central.uh.edu>
handle the situation where the user requests a non-zero amount
of data but has a zero-size fileview. My instrinct would have been
to return an error code, but according to the test that I used
it should be MPI_SUCCESS and zero bytes. It is definitely better
than segfaulting :-)
THis makes another test from the IBM testsuite pass.
Signed-off-by: Edgar Gabriel <egabriel@central.uh.edu>
the seek offset calculation did not treat the offset as a multiple
of the etype provided. Fixing this makes some more ibm tests pass.
Signed-off-by: Edgar Gabriel <egabriel@central.uh.edu>
