Commit fec519a793 broke the ability to
run autogen.pl in a distribution tarball. This commit restores that
ability by also distributing opal/mca/hwloc/autogen.options in the
tarball.
Skipping CI because CI does not test this functionality:
[skip ci]
bot:notest
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Several component-specific functions were named with a prefix of
"opal_timer_base", which was quite confusing. Rename them to have a
prefix "opal_timer_linux" to make it clear that they are here in this
component (and different than *actual* opal_timer_base symbols).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
This commit makes datagram checks time based and reduces their
frequency when only the wildcard datagram is posted. This change
improves latency on knl systems.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit removes the local operation count check from the local SMSG
completion queue. This check was leading to hangs due to an undocumented
feature of the ugni library. The local SMSG CQ is used to send credit
return messages back to the sender. The ugni library never checks for
the completion itself but relying on the SMSG user to periodically
check the CQ.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit updates the ugni btl to make use of multiple device
contexts to improve the multi-threaded RMA performance. This commit
contains the following:
- Cleanup the endpoint structure by removing unnecessary field. The
structure now also contains all the fields originally handled by the
common/ugni endpoint.
- Clean up the fragment allocation code to remove the need to
initialize the my_list member of the fragment structure. This
member is not initialized by the free list initializer function.
- Remove the (now unused) common/ugni component. btl/ugni no longer
need the component. common/ugni was originally split out of
btl/ugni to support bcol/ugni. As that component exists there is no
reason to keep this component.
- Create wrappers for the ugni functionality required by
btl/ugni. This was done to ease supporting multiple device
contexts. The wrappers are thread safe and currently use a spin
lock instead of a mutex. This produces better performance when
using multiple threads spread over multiple cores. In the future
this lock may be replaced by another serialization mechanism. The
wrappers are located in a new file: btl_ugni_device.h.
- Remove unnecessary device locking from serial parts of the ugni
btl. This includes the first add-procs and module finalize.
- Clean up fragment wait list code by moving enqueue into common
function.
- Expose the communication domain flags as an MCA variable. The
defaults have been updated to reflect the recommended setting for
knl and haswell.
- Avoid allocating fragments for communication with already
overloaded peers.
- Allocate RDMA endpoints dyncamically. This is needed to support
spreading RMA operations accross multiple contexts.
- Add support for spreading RMA communication over multiple ugni
device contexts. This should greatly improve the threading
performance when communicating with multiple peers. By default the
number of virtual devices depends on 1) whether
opal_using_threads() is set, 2) how many local processes are in the
job, and 3) how many bits are available in the pid. The last is
used to ensure that each CDM is created with a unique id.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit exposes ugni statistics for use with MPI_T. There is
no overhead to providing these counters.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
hwloc v1.5 does not support HWLOC_OBJ_OSDEV_COPROC
nor hwloc_topology_dup(), so for this version :
- do not search for coprocessors
- do not try hwloc_topology_dup(), note this is not
used anywhere in the code base
Thanks Jeff for helping with the wording
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
This is mostly for error cases, where we need to release the
newly created proc. Currently the code deadlocks because the endpoint
lock is help at the release and the lock is not recursive.
Aslo added some code to print the IP addresses that don't match during
the TCP connection step.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Per a prior commit, the presence of "hwloc.h" can cause ambiguity when
using --with-hwloc=external (i.e., whether to include
opal/mca/hwloc/hwloc.h or whether to include the system-installed
hwloc.h).
This commit:
1. Renames opal/mca/hwloc/hwloc.h to hwloc-internal.h.
2. Adds opal/mca/hwloc/autogen.options to tell autogen.pl to expect to
find hwloc-internal.h (instead of hwloc.h) in opal/mca/hwloc.
3. s@opal/mca/hwloc/hwloc.h@opal/mca/hwloc/hwloc-internal.h@g in the
rest of the code base.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
* Include a 'demo' component that shows some of the features.
* Currently has hooks for:
- MPI_Initialized
- top, bottom
- MPI_Init_thread
- top, bottom
- MPI_Finalized
- top, bottom
- MPI_Init
- top (pre-opal_init), top (post-opal_init), error, bottom
- MPI_Finalize
- top, bottom
* Other places in ompi can 'register' to hook into any one of these places
by passing back a component structure filled with function pointers.
* Add a `MCA_BASE_COMPONENT_FLAG_REQUIRED` flag to the MCA structure that
is checked by the `hook` framework. If a required, static component has
been excluded then the `hook` framework will fail to initialize.
- See note in `opal/mca/mca.h` as to why this is checked in the `hook`
framework and not in `opal/mca/base/mca_base_component_find.c`
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
in this context, AMD64 really means amd64 or em64t, so let's
rename this into X86_64 in order to avoid any confusion
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
Return value in comment about opal_list_item_compare_fn_t typedef when a < b is indicated to be 11 instead of -1.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
This commit makes the vma tree garbage collection list a lifo. This
way we can avoid having to hold any lock when releasing vmas. In
theory this should finally fix the hold-and-wait deadlock detailed
in #1654.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Allow someone to specify the "pe=N" modifier to a mapping policy when N=1. This equates to just "bind-to core", but helps people who use a script to set the PE policy. Fix a bug where setting the binding policy left a lingering "if-supported" flag that shouldn't be there.
Signed-off-by: Ralph Castain <rhc@open-mpi.org>
Cleanup a race condition segfault during finalize by ensuring the PMIx progress thread is stopped prior to starting to tear down the messaging components
Signed-off-by: Ralph Castain <rhc@open-mpi.org>
Do not use opal_output_verbose inside O(n) loops. This was causing us
to make O(n) calls to snprintf which was greatly slowing launch at
scale.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds a new base enumerator type for variables that take of
the values -1, 0, and 1. These values are mapped to the strings auto,
false, true. This commit updates the mpi_leave_pinned MCA variable to
use the new enumerator.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Add a verbose to show all the failed attempts to match the
remote interfaces based on the modex info.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
It's possible that we can have zlib.h but still not have zlib support.
Use the correct macro to protect the usage of calling zlib functions.
This fixes 32-bit MTT builds at Cisco (e.g.,
https://mtt.open-mpi.org/index.php?do_redir=2389).
Submitted upstream to PMIX: https://github.com/pmix/master/pull/290
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
PMIx_server_register_nspace() is an asynchronous operation, so
the pmix glue wait for it completes before returning.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
the argc field from the opal_pmix_app_t struct was removed,
so adjust the pmix/ext11 glue accordingly.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
- New MCA option: opal_stacktrace_output
- Specifies where the stack trace output stream goes.
- Accepts: none, stdout, stderr, file[:filename]
- Default filename 'stacktrace'
- Filename will be `stacktrace.PID`, or if VPID is available,
then the filename will be `stacktrace.VPID.PID`
- Update util/stacktrace to allow for different output avenues
including files. Previously this was hardcoded to 'stderr'.
- Since opal_backtrace_print needs to be signal safe, passing it a
FILE object that actually represents a file stream is difficult. This
is because we cannot open the file in the signal handler using
`fopen` (not safe), but have to use `open` (safe). Additionally, we
cannot use `fdopen` to convert the `int fd` to a `FILE *fh` since it
is also not signal safe.
- I did not want to break the backtrace.h API so I introduced a new
rule (documented in `backtrace.c`) that if the `FILE *file`
argument is `NULL` then look for the `opal_stacktrace_output_fileno`
variable to tell you which file descriptor to use for output.
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
Since the oob and connections systems do not work the same way they
did in older versions of Open MPI these operations are no longer
necessary. At best they do nothing and at worst they hurt performance
by making us enter the event library more often in opal_progress().
Fixes#2839
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
When Java bindings are used, MPI_Init() is not invoked
by the main thread, and this causes some keys being destructed twice.
Reset the per thread values to NULL in order to correctly handle this
Fixesopen-mpi/ompi#2811
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
* Similar to `orte_map_stddiag_to_stderr` except it redirects `stddiag`
to `stdout` instead of `stderr`.
* Add protection so that the user canot supply both:
- `orte_map_stddiag_to_stderr`
- `orte_map_stddiag_to_stdout`
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
- This allows the following MCA option to have an impact on the
framework verbose output as well.
* `-mca mca_base_verbose stdout`
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
* Adds a parameter to adjust the method used by libevent.
- Matches that of the libevent2022 component.
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
Note: since the discovered cpus are filtered against this list, #slots will be set to the #cpus in the list if no slot values are given in a -host or -hostname specification.
Signed-off-by: Ralph Castain <rhc@open-mpi.org>
* The AC_LANG_PROGRAM macro adds the `main()` so it is erroneous
to add it to the test program.
* This was detected with the XL compilers which will fail to
build the program in this situation. The GNU compiler does not
error out or warn, but successfully compiles the program.
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
always check the permissions of the created directory,
in case some one else created the very same directory but
with incompatible permissions
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
Due to the conversion from ssize_t to int we were losing bytes, and
ended up writing outside the receiver buffer. Similarly on the send,
due to the conversion to a lesser type, we could missinterpret the
end of the fragment.
The linux timer code was multiplying the result of the x86 time stamp
counter by 1000000 before dividing by the cpu frequency. This can
cause us to overflow 64 bits if the time stamp counter grows larger
than ~ 1.8e13 (about 8400 seconds after boot). To fix the issue the
units of opal_timer_linux_freq have been changed to MHz.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
the hwloc topology might not contain a NUMA object with hwloc < v2
if the node is not NUMA, so force the NUMA object count to one
in order to correctly allocate mca_btl_sm_component.sm_mpools.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
Add a missing constraint to the input operand list.
This fixes a regression caused by d4be138a7b.
Thanks to Orion Poplawski for reporting the issue.
Refs #2610
Signed-off-by: Nysal Jan K.A <jnysal@in.ibm.com>
This commit fixes a deadlock that can occur when the libc version
holds a lock when calling munmap. In this case we could end up calling
free() from vma_tree_delete which would in turn try to obtain the lock
in libc. To avoid the issue put any deleted vma's in a new list on the
vma module and release them on the next call to vma_tree_insert. This
should be safe as this function is not called from the memory hooks.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This should probably not go to the v2.x branch, since it changes the
output format of the usnic stats.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Double check the queue lengths that we get back from libfabric to
ensure that they are at least as long as we need. They *should* never
be shorter than we need, but let's just check to be sure.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Don't just blindly send ACKs; ensure that we have send credits before
doing so. If we don't have any send credits, just don't send the ACK
(it'll come again soon enough; it's not a tragedy if we don't send it
now).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
The libfabric usnic provider may give you back TX/RX queues that are
longer than you asked for. So just use the TX/RQ/CQ lengths that we
asked for, regardless of what length comes back.
Additionally, keep the length of the priority channel CQ separate from
the length of the data CQ.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Since the usnic BTL is single-threaded in this area, there really is
no danger, but don't use one of the pointers hanging off the frag
after we return it to the freelist. Instead, save the endpoint
pointer before returning the frag.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
The types are technically typedef equivalent, but it's less confusing
to use the types that agree with the name of the constructor.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
on Linux, sending the header and then the message data does severely
impact performances of ptl/tcp :
on the receiver, reading the data can often result in an PMIX_ERR_RESOURCE_BUSY
or PMIX_ERR_WOULD_BLOCK, which ends up degrading performances)
this commit send both header and message data at the same time via writev()
and makes ptl/tcp virtually as efficient as ptl/usock.
Short writev generally occur when the kernel buffer is full, so there is no
point for retrying in this case.
fwiw, no such degradation was observed on OSX.
Refs open-mpi/ompi#2657
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
The MCA variable code calls the string from value function with a NULL
string to verify values. The verbosity enumerator was not correctly
checking for a non-NULL value before trying to set the string.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
