This has shown to be more effective in achieving overlap
of inter- and intra-node communication and reduces the inital
delay before hitting the network.
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Also make coll/tuned the default for shared memory communication
as coll/sm has shown performance issues that need investigation.
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
The selectable list is sorted with lowest to highest priority so the
user-defined preferences should be appended to the list.
The preference treatment should also maintain the order provided by the user
(first item has highest priority) so switch the loop order.
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
- Add some missing AC_CHECK_SIZEOF's in configure.ac
- Remove some unused variables
- Initialize some variables
- Fix some parameter types
- Cast where appropriate/safe to fix warnings
- Move ompi/mca/common/monitoring Fortran bindings to a separate .c
file so that they can use different #define's than the C bindings,
and therefore compile properly / without warnings.
- Fix signedness discrepancies
- Who knew? Separated these into multiple #if's, instead:
```
// This is undefined behavior
#define HAVE_FOO defined(FOO)
#define YOW (HAVE_FOO && defined(BAR))
```
- Fix some typos in OMPI_BUILD_HOST logic
- Don't "2>/dev/null" in OMPI_BUILD_HOST logic; it just hides errors
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
The total size depends on number of ranks so the usual ranges don't work.
Thus, use the average across all ranks to make a decision.
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
MPI_Ialltoallw() and friends take a const MPI_Datatype types[] argument.
In order to be able to call OBJ_RELEASE(types[0]), we used to simply
drop the const modifier. This change make it right by introducing the
OBJ_RELEASE_NO_NULLIFY(object) macro that no more set object = NULL
if the object is freed.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
These selections seem harmful in my measurements and don't seem to be
motivated by previous measurement data.
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Bcast: scatter_allgather and scatter_allgather_ring expect N_elem >= N_procs
Allreduce: rabenseifner expects N_elem >= pow2 nearest to N_procs
In all cases, the implementations will fall back to a linear implementation,
which will most likely yield the worst performance (noted for 4B bcast on 128 ranks)
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
The mca parameters coll_tuned_*_algorithm are ignored unless coll_tuned_use_dynamic_rules is true so mention that in the description.
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Among many other things:
- Fix an imbalance bug in MPI_allgather
- Accept more human readable configuration files. We can now specify
the collective by name instead of a magic number, and the component
we want to use also by name.
- Add the capability to have optional arguments in the collective
communication configuration file. Right now the capability exists
for segment lengths, but is yet to be connected with the algorithms.
- Redo the initialization of all HAN collectives.
Cleanup the fallback collective support.
- In case the module is unable to deliver the expected result, it will fallback
executing the collective operation on another collective component. This change
make the support for this fallback simpler to use.
- Implement a fallback allowing a HAN module to remove itself as
potential active collective module, and instead fallback to the
next module in line.
- Completely disable the HAN modules on error. From the moment an error is
encountered they remove themselves from the communicator, and in case some
other modules calls them simply behave as a pass-through.
Communicator: provide ompi_comm_split_with_info to split and provide info at the same time
Add ompi_comm_coll_preference info key to control collective component selection
COLL HAN: use info keys instead of component-level variable to communicate topology level between abstraction layers
- The info value is a comma-separated list of entries, which are chosen with
decreasing priorities. This overrides the priority of the component,
unless the component has disqualified itself.
An entry prefixed with ^ starts the ignore-list. Any entry following this
character will be ingnored during the collective component selection for the
communicator.
Example: "sm,libnbc,^han,adapt" gives sm the highest preference, followed
by libnbc. The components han and adapt are ignored in the selection process.
- Allocate a temporary buffer for all lower-level leaders (length 2 segments)
- Fix the handling of MPI_IN_PLACE for gather and scatter.
COLL HAN: Fix topology handling
- HAN should not rely on node names to determine the ordering of ranks.
Instead, use the node leaders as identifiers and short-cut if the
node-leaders agree that ranks are consecutive. Also, error out if
the rank distribution is imbalanced for now.
Signed-off-by: Xi Luo <xluo12@vols.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@icl.utk.edu>
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* first import of Bull specific modifications to HAN
* Cleaning, renaming and compilation fixing Changed all future into han.
* Import BULL specific modifications in coll/tuned and coll/base
* Fixed compilation issues in Han
* Changed han_output to directly point to coll framework output.
* The verbosity MCA parameter was removed as a duplicated of coll verbosity
* Add fallback in han reduce when op cannot commute and ppn are imbalanced
* Added fallback wfor han bcast when nodes do not have the same number of process
* Add fallback in han scatter when ppn are imbalanced
+ fixed missing scatter_fn pointer in the module interface
Signed-off-by: Brelle Emmanuel <emmanuel.brelle@atos.net>
Co-authored-by: a700850 <pierre.lemarinier@atos.net>
Co-authored-by: germainf <florent.germain@atos.net>
a hierarchical, architecture-aware collective communication module.
Add Reduce and remove up_seg_size and low_seg_size in Bcast
Increase HAN's priority
Signed-off-by: Xi Luo <xluo12@vols.utk.edu>
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
- Add support for fallback to previous coll module on non-commutative operations (#30)
- Replace mutexes by atomic operations.
- Use the correct nbc request type (for both ibcast and ireduce)
* coll/base: document type casts in ompi_coll_base_retain_*
- add module-wide topology cache
- use standard instead of synchronous send and add mca parameter to control mode of initial send in ireduce/ibcast
- reduce number of memory allocations
- call the default request completion.
- Remove the requests from the Fortran lookup conversion tables before completing
and free it.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Joseph Schuchart <schuchart@hlrs.de>
Co-authored-by: Joseph Schuchart <schuchart@hlrs.de>
Reduce scatter block and reduce scatter algorithms were hitting
correctness issues for non commutative strided tests. We will revert to
the original default algorithms for those two collectives (basic linear
and non overlapping respectively) in the non commutative op case.
See #8010
Signed-off-by: William Zhang <wilzhang@amazon.com>
As it is possible to have multiple outstanding non-blocking collectives
provided by different collective modules, we need a consistent
mechanism to allow them to select unique tags for each instance of a
collective.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* piggybacking Bull functionalities
* coll/adapt: Fix naming conventions and C11 atomic use
This commit fixes some naming convention issues, such as function names
which should follow the naming ompi_coll_adapt instead of
mca_coll_adapt, reserved for component and module naming (cf. tuned
collective component);
It also fixes the use of _Atomic construct, which is only valid in C11.
OPAL constructs have already been adapted to that use, so use
opal_atomic_* types instead.
* coll/adapt: Remove unused component field in module
This commit removes an unneeded field referencing the component in the
module of adapt, as it is already available through the
mca_coll_adapt_component global variable.
Signed-off-by: Marc Sergent <marc.sergent@atos.net>
Co-authored-by: Lemarinier, Pierre <pierre.lemarinier@atos.net>
Co-authored-by: pierrele <31764860+pierrele@users.noreply.github.com>
The default algorithm selections were out of date and not performing
well. After gathering data from OMPI developers, new default algorithm
decisions were selected for:
allgather
allgatherv
allreduce
alltoall
alltoallv
barrier
bcast
gather
reduce
reduce_scatter_block
reduce_scatter
scatter
These results were gathered using the ompi-collectives-tuning package
and then averaged amongst the results gathered from multiple OMPI
developers on their clusters.
You can access the graphs and averaged data here:
https://drive.google.com/drive/folders/1MV5E9gN-5tootoWoh62aoXmN0jiWiqh3
Signed-off-by: William Zhang <wilzhang@amazon.com>
The gather and scatter operations did not use the correct message size
(Only did datatype size * com size). This did not correctly reflect the
total message size and prevents fine tuning within a com size. This
patch multiplies the value by the number of elements sent.
Signed-off-by: William Zhang <wilzhang@amazon.com>
