The add_procs change made some assumptions in the bml/r2 add_procs
wrong. This lead to del_procs never being called. I removed the logic
that checks the ompi_proc_t reference count and removed an unnecessary
allocation. The allocation only makes sense if we pass more than a
single proc at a time to the btl del_procs.
This commit also ensures that the btl del_procs is called if the
endpoint is in the btl_rdma array but not the btl_send array.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The osc/sm component was using a simple counter to determine if all
expected posts had arrived to start a PSCW access epoch. This is
incorrect as a post may arrive from a peer that isn't part of the
current start group. There are many ways this could have been fixed.
This commit adds an n^2 bitmap. When a process posts it sets a bit in
the bitmap associated with the access rank to indicate the post is
complete. The access rank checks for and clears the bits associated
with all the processes in the start group.
The bitmap requires comm_size ^ 2 bits of space. This should be
managable as most nodes have relatively small numbers of processes. If
this changes another algorigthm can be implemented.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Fix CID 1324733: Null pointer dereferences (FORWARD_NULL)
Fix CID 1324734: Null pointer dereferences (FORWARD_NULL)
Fix CID 1324735: Null pointer dereferences (FORWARD_NULL)
Fix CID 1324736: Null pointer dereferences (FORWARD_NULL)
Fix CID 1324737: Null pointer dereferences (FORWARD_NULL)
Fix CID 1324751: Memory - illegal accesses (USE_AFTER_FREE)
Fix CID 1324750: (USE_AFTER_FREE)
Fix CID 1324749: Memory - corruptions (USE_AFTER_FREE)
Fix CID 1324748: Memory - illegal accesses (USE_AFTER_FREE)
Fix CID 1324747: (USE_AFTER_FREE)
Fix CID 1324746: Memory - corruptions (USE_AFTER_FREE)
Add missing return on an error path.
Fix CID 1324745: Code maintainability issues (UNUSED_VALUE)
Ignore return code from barrier. It was not being used anyway.
Fix CID 1324738: Null pointer dereferences (FORWARD_NULL)
Fix CID 1324741: Null pointer dereferences (REVERSE_INULL)
module->selected_btl can not be NULL in osc/rdma during normal
operation. Removed the unnecessary NULL check.
Fix CID 1324752: Memory - illegal accesses (USE_AFTER_FREE)
Move ompi_osc_pt2pt_module_lock_remove to before the lock is freed.
Fix CID 1324744: Uninitialized variables (UNINIT)
Fix CID 1324743: Uninitialized variables (UNINIT)
This array is not used unitialized but there is no reason not to use
calloc here to silence the warning.
The following CID is a false positive: 1324742. I will mark it such in
coverity.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds support for performing one-sided operations over
supported hardware (currently Infiniband and Cray Gemini/Aries). This
component is still undergoing active development.
Current features:
- Use network atomic operations (fadd, cswap) for implementing
locking and PSCW synchronization.
- Aggregate small contiguous puts.
- Reduced memory footprint by storing window data (pointer, keys,
etc) at the lowest rank on each node. The data is fetched as each
process needs to communicate with a new peer. This is a trade-off
between the performance of the first operation on a peer and the
memory utilization of a window.
TODO:
- Add support for the accumulate_ops info key. If it is known that
the same op or same op/no op is used it may be possible to use
hardware atomics for fetch-and-op and compare-and-swap.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit updates osc/pt2pt to allocate peer object as they are
needed rather than all at once. Additionally, to help improve the
memory footprint a new synchronization structure has been added.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Add an accessor for the proc_endpoints[OMPI_PROC_ENDPOINT_TAG_MTL]
member of the ompi_proc_t structure. This accessort calls add_procs
with the ompi_proc_t if the member is NULL.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
Add an accessor for the proc_endpoints[OMPI_PROC_ENDPOINT_TAG_MTL]
member of the ompi_proc_t structure. This accessort calls add_procs
with the ompi_proc_t if the member is NULL. Tested on an infinipath
system with InfiniPath_QLE7340 HCAs.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
This commit modifies the ompi_group_t union/difference code to compare/copy the
raw group values. This will either be a ompi_proc_t or a sentinel value. This
commit also adds helper functions to convert between opal process names and
sentinel values.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
This commit contains the following changes:
- pml/ob1: use the bml accessor function when requesting a bml
endpoint. this will ensure that bml endpoints are only created when
needed. for example, a bml endpoint is not requested and not
allocated when receiving an eager message from a peer.
- pml/ob1: change the pml_procs array in the ob1 communicator to a
proc pointer array. at the cost of a single level of extra
redirection this will allow us to allocate pml procs on demand.
- pml/ob1: add an accessor function to access the pml proc structure
for a given peer. this function will allocate the proc if it
doesn't already exist.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit contains the following changes:
- bml: add a function to add a single process. this function is
intended to remove the need to maintain a opal_bitmap_t as it is
irrelevant for a single proc. BTLs will need to be updated to
either 1) ignore the return code from opal_bitmap_set_bit or not
call the function if the reachability bitmap is NULL.
- bml: add an inline accessor function for getting the bml endpoint
for a peer proc. this function will either 1) return the cached bml
endpoint, or 2) create the endpoint and call add_proc will all
available BTL modules.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
After long debugging, I found last week the reason this optimization originally broke
some hdf5 tests. We now pass the hdf5 test suite with the optimization being actively used.
Specifically:
- reduce the number of realloc's and malloc's by moving
some arrays out of the cycle loop, if we know that there
size is not changing
- store the rank of the aggregator in a separate variable to avoid
continuous dereferencing
- change the wait_all logic in write_all to use a fix number of requests
(even if they are MPI_REQUEST_NULL)
- fix the timing to considere the two initial allgather and the one
allgatherv operation to be a part of it
- add more comments.
- MPI_Compare_and_swap
- MPI_Fetch_and_op
- MPI_Raccumulate
- MPI_Win_detach
Thanks to Michael Knobloch and Takahiro Kawashima for bringing this
to our attention
Bring Slurm PMI-1 component online
Bring the s2 component online
Little cleanup - let the various PMIx modules set the process name during init, and then just raise it up to the ORTE level. Required as the different PMI environments all pass the jobid in different ways.
Bring the OMPI pubsub/pmi component online
Get comm_spawn working again
Ensure we always provide a cpuset, even if it is NULL
pmix/cray: adjust cray pmix component for pmix
Make changes so cray pmix can work within the integrated
ompi/pmix framework.
Bring singletons back online. Implement the comm_spawn operation using pmix - not tested yet
Cleanup comm_spawn - procs now starting, error in connect_accept
Complete integration
Portals4 supports atomic ops on datatypes less than or equal to
max_fetch_atomic_size bytes. This commit fixes a bug that required
the datatype to be less than max_fetch_atomic_size bytes.
- make the internal structure follow the Open MPI naming convention
- provide a single flag/macro which controls the compilation/utilization of this
feature, to avoid that somebody using this has to modify every single
fcoll component. A configure option could be added later if desired.
configury: fix hcoll, fca and mxm detection and revamp yalla Makefile.am
Thanks to David Shrader and Ake Sandgren for bringing this issue to our attention
* do not add -I/.../include/fca -I /.../include/fca_core to CPPFLAGS
* allow configure --with-fca
* search fca libs in both DIR/lib and DIR/lib64
* fix the description of the --with-fca option
* do not add -I/.../include/hcoll -I /.../include/hcoll/api to CPPFLAGS
* allow configure --with-hcoll
* search hcoll libs in both DIR/lib and DIR/lib64
* fix the description of the --with-hcoll option
mtl_ofi_provider_include (resp. mtl_ofi_provider_exclude) can be used
to specify which provider(s) the OFI MTL can select (resp. ignore).
e.g. --mca mtl_ofi_provider_include "psm,sockets"
By default, mtl_ofi_provider_exclude is set to "sockets,mxm".
This deprecates the old MCA var named "mtl_ofi_provider".
This commit does the following:
* s/ompi_check_treematch/ompi_topo_treematch/ (i.e., abide by the
prefix rule)
* change the value of ompi_topo_treematch_happy from yes/no to 0/1, so
that we can use -eq for numerical comparisons (vs. string
comparisons). It's the little things in life, no?
* Check the valueo f $OPAL_HAVE_HWLOC to ensure that hwloc support is
enabled. If not, disqualify treematch from building.
* Fixes a few places that were underquoted
* Convert from "test ... -a ..." to "test ... && test ..."
Fixesopen-mpi/ompi#797
This commit rewrites parts of libnbc to fix issues identified by
coverity and myself. The changes are as follows:
- libnbc function would return invalid error codes (internal to
libnbc) to the mpi layer. These codes names are of the form
NBC_. They do not match up with the error codes expected by the mpi
layer. I purged the use of all these error codes with the exception
of NBC_OK and NBC_CONTINUE in progress. These codes are used to
identify when a request handle is complete.
- Handles and schedules were leaked by all collective routines on
error. A new routine was added to return a collective handle
(NBC_Return_handle).
- Temporary buffers containting in/out neighbors for neighborhood
collectives were always leaked.
- Neigborhood collectives contained code to handle MPI_IN_PLACE which
is never a valid input for the send or receive buffer. Stipped this
code out.
- Files were inconsistently named. Most are nbc_isomething.c but one
was named coll_libnbc_ireduce_scatter_block.c.
- Made the NBC_Schedule "structure" and object so it can be
retained/released. This may enable the use of schedule caching at a
later time. More testing will be needed to ensure the caching code
works. If it doesn't the code should be stripped out completely.
- Added code to simply common case of scheduling send/recv +
barrier.
- Code cleanup for readability.
The code now passes the clang static analyzer.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Some OFI providers such as "sockets" are used for debugging
purposes mostly. For these providers, other components usually
offer better performance -- e.g. for sockets, the BTL/TCP would
be a better choice.
Thus, we chose to ignore some providers unless explicitly asked
by the user on the command line:
e.g. --mca mtl_ofi_provider sockets
When configured with --enable-picky
topo_base_lazy_init.c compiles with a warning:
CC base/topo_base_lazy_init.lo
base/topo_base_lazy_init.c:46:67: warning: implicit conversion from enumeration type 'enum mca_base_register_flag_t' to different enumeration type 'mca_base_open_flag_t' (aka 'enum mca_base_open_flag_t') [-Wenum-conversion]
err = mca_base_framework_open (&ompi_topo_base_framework, MCA_BASE_REGISTER_DEFAULT);
This commit fixes this implicit conversion problem.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
our optimized default file view. Otherwise, performance will suffer. file_get_view should still return the correct filetype, not our optimized default file view
- some application use MPI_File_delete as a collective function (e.g. IOR), which I think is not really covered by the standard. Right now, one process succeeds and theother ones return an error code. Fix that by not returning no error if the file that we try to delete does not exist anymore, to make these applications work.
Retain inline progress function for ofi
mtl, but have a non-inlined progress function
which is registered with the opal progress
mechanism.
@jithinjosepkl
I've bad news about the psm provider. I still notice
segfaults - not always - but frequently at finalize
when using the psm provider. I don't notice this
when using the sockets provider.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
at Inria Bordeaux. This allows us to take advantage of the remap
capability of MPI to rearrange the ranks beased on the weights
povided by the application.
Fix the indentation and protect with __DEBUG__ one fprintf.
Add the Cecill-B license to the imported library.
Fix a compiler warning.
Restrict the TreeMatch dependencies.
The TreeMatch software is released under BSD3 (as indicated by their
copyright information @
https://gforge.inria.fr/scm/viewvc.php/COPYING?view=markup&root=treematch).
Update the README.
Even if the mutex is actually located in
sm_data->sm_offset_ptr->mutex, have sm_data->mutex point to it. This
avoids a few #if blocks that are otherwise identical.
This commit does two things. It removes checks for C99 required
headers (stdlib.h, string.h, signal.h, etc). Additionally it removes
definitions for required C99 types (intptr_t, int64_t, int32_t, etc).
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
This commit fixes several bugs in the static request objects used by
ob1 for blocking send/receive operations.
- Fix memory leak when using MPI_THREAD_MULTIPLE. Requests were
allocated off the free list but were destructed and NOT returned.
- Fix double-destruct of static objects. There is no reason to
CONSTRUCT/DESTUCT the static object for each send/receive
operation. This adds overhead and no benefit. To keep the code
clean helper functions have been added to finalize ob1 send/receive
requests.
- Remove now unnecessary include of alloca.h.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
This new MTL runs over PSM2 for Omni Path. PSM2 is a descendant of PSM
with changes to support more ranks and some MPI-3 features like mprobe.
PSM2 will only support Omni Path networks; PSM only supports True Scale.
Likewise, the existing PSM MTL will continue to be maintained for True
Scale, while the PSM2 MTL is developed and maintained for Omni Path.
from the message queues (a debugging feature). With this approach
all blocking (single threaded) requests are allocated from the main
freelist, so they will be accounted for during the message queues
investigation).
The Portals4 MTL allocates two Portals IDs requesting specific
well-known IDs and assumes that those IDs are allocated. If those IDs
are in use, PtlPTAlloc() will allocate a different ID. This commit
verifies that the requested IDs were allocated.
CID 1295340 Unchecked return value (CHECKED_RETURN)
Check the return code of mca_base_framework_open. If the call fails for some reason
the component array will not be properly defined. This will cause issues in
mca_topo_base_find_available.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
