This commit reduces the overhead of calling the ugni progress
function. It does the following:
- Check for new connections once every eight calls.
- Do not call remote smsg progress unless we are connected to at
least one remote peer.
- Do not call rdma progress unless at least one rdma fragment is
outstanding.
- Check endpoint wait list size before obtaining a lock.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit reduces the default exclusivity so that btl/scif is not
used for send/recv over other shared memory transports.
Fixesopen-mpi/ompi#1712
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Our checks and the ones of libevent are somewhat flawed.
If adding multiple "-framework" to CXXFLAGS or CFLAGS, we strip
the keyword from the command-line, not good.
libevent however assumes plain gcc without testing properly
that the compiler supports -Wno-deprecated-declarations.
* Remodel the request.
Added the wait sync primitive and integrate it into the PML and MTL
infrastructure. The multi-threaded requests are now significantly
less heavy and less noisy (only the threads associated with completed
requests are signaled).
* Fix the condition to release the request.
The OPAL_CMA_NEED_SYSCALL_DEFS is always defined/set to 0 or 1. Therefore
instead of checking if the macro is defined, we have to look at the value
itself.
Signed-off-by: Pavel Shamis (Pasha) <pasharesearch@gmail.com>
This commit changes the behavior of bml/r2 from conditionally
registering btl progress functions to always registering progress
functions. Any progress function beloning to a btl that is not yet in
use is registered as low-priority. As soon as a proc is added that
will make use of the btl is is re-registered normally.
This works around an issue with some btls. In order to progress a
first message from an unknown peer both ugni and openib need to have
their progress functions called. If either btl is not in use after the
first call to add_procs the callback was never happening. This commit
ensures the btl progress function is called at some point but the
number of progress callbacks is reduced from normal to ensure lower
overhead when a btl is not used. The current ratio is 1 low priority
progress callback for every 8 calls to opal_progress().
Fixesopen-mpi/ompi#1676
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Take another shot at untangling the spaghetti
orterun: fix for command line parsing
orte-submit calls opal_init_util () before parsing out MCA command line
options (-mca, -am, etc). This prevents mpirun from setting opal MCA
variables for some frameworks as well as the MCA base. This is because
when a framework is opened all of its variables are set to read-only.
Eventually we want to lift this restriction on some MCA variables but
since -mca is affected we must parse out the MCA command line options
before opal_init_util(). This commit fixes the bug by adding a new
option to opal_cmd_line_parse (ignore unknown option) so orte-submit
can pre-parse the command line for MCA options.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
Minor cleanups to avoid releasing/recreating the cmd line
Remove contrib/windows/
Merge hwlocXYZ/hwloc/README-ompi.txt back into hwlocXYZ/README-ompi.txt instead of having both.
Add README.txt in new automake-required directory contrib/systemd/
Keep the following patches applied since they are not in 1.11.3
linux: actually enable libudev based on the result of AC_CHECK_LIB
(cherry picked from open-mpi/hwloc@9549fd59af)
configure: check the actual may_alias syntax that we use
(cherry picked from open-mpi/hwloc@0ab7af5e90)
This commit fixes several bugs in the registration cache code:
- Fix a programming error in the grdma invalidation function that can
cause an infinite loop if more than 100 registrations are
associated with a munmapped region. This happens because the
mca_rcache_base_vma_find_all function returns the same 100
registrations on each call. This has been fixed by adding an
iterate function to the vma tree interface.
- Always obtain the vma lock when needed. This is required because
there may be other threads in the system even if
opal_using_threads() is false. Additionally, since it is safe to do
so (the vma lock is recursive) the vma interface has been made
thread safe.
- Avoid calling free() while holding a lock. This avoids race
conditions with locks held outside the Open MPI code.
Fixesopen-mpi/ompi#1654.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Update external as well
Revise the change: we still need the MPI_Barrier in MPI_Finalize when we use a blocking fence, but do use the "lazy" wait for completion. Replace the direct logic in MPI_Init with a cleaner macro
WRAPPER_EXTRA flags are checked *before* the POST_CONFIG macro is
invoked. So set them in the main CONFIG macro.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
This commit adds an additional check for 64-bit atomic support for __sync
builtins. If 64-bit support is not available the opal_atomic_*_64 atomics
are disabled.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
xlc 13.1.0 crashes because of our may_alias attributes in nolibxml.c
on Power7. libxml.c and nolibxml.c are the only may_alias users for now,
so change our configure check to match the actual code using it.
Thanks to Paul Hargrove for reporting and debugging the issue,
and providing the patch.
https://www.open-mpi.org/community/lists/devel/2016/05/18918.php
(cherry picked from open-mpi/hwloc@0ab7af5e90)
Per discussion on the mailing list and with IBM, remove the AIX timer
code (since AIX is no longer supported).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
This commit disables the linux patcher component due to a limitation
in loader patching. While this component is effective in patching
calls made within Open MPI and by the application it fails to hook
calls made within glibc. This means the munmap call made by free is
not correctly hooked. Until this problem can be resolved this
component will remain disabled. If it can't be resolved this component
should probably be removed.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The table of contents (TOC) code only appears to only apply to
ppc64. The code was incorrectly assuming the existence of the TOC on
ppc32. This commit updates the necessary code to only apply to ppc64.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes a compile/link issue caused by vader. The vader btl
was using OPAL_THREAD_ADD64 to increment a counter which may not be
available on 32-bit systems. Changed to use OPAL_THREAD_ADD_SIZE_T
which will be 64-bit or 32-bit depending on the system.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
ebx can not be clobbered when using -fPIC so save and restore the
register instead of allowing it to be clobbered.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
instead of doing AC_CHECK_HEADERS+AC_CHECK_LIB and only using the result of the former.
Thanks to Paul Hargrove for reporting the issue (OMPI build with -m32).
(cherry picked from open-mpi/hwloc@9549fd59af)
This commit fixes a compilation issue with some versions of exp
verbs. In some cases struct ibv_exp_device_attr does not have either
the exp_atom or exp_atomic_cap fields. It is fine to drop one check
and fall back to the non-exp attribute check on the other.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Add a feature check for clflush before trying to use the clflush
instruction. As far as I can tell there is no equivalent before the
SSE2 instruction set.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The function signature of mremap on BSD (NetBSD, FreeBSD) differs from
the linux version. Added support for the BSD style of mremap.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes a compile error when the system has mremap but not
MREMAP_FIXED. In this case we do not care about the value of
new_address as the argument does not exist.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
In some rare cases when a process receives the connect ack while
locally updating the peer endpoint structure, we could drop the
incomming connect ack due to the fact that the send handler is
protected with a try lock (on the endpoint) and our initial send
event was not persistent. Making the send event persistent solves
all issues.
Rather than have a stub function for the pmix fence_nb
operation, just set to NULL. Causes fewer problems.
Fixes#1597Fixes#1527
Signed-off-by: hppritcha <howardp@lanl.gov>
This commit fixes a race between a thread calling the tcp btl's
add_procs and a thread processing an incomming connection. The race
occured because the add_procs thread adds a newly created proc object
to the hash table *before* the object is fully initialized. The
connection thread then attempts to use the object before the endpoints
array on the object has beeen allocation. The fix is to only add the
proc to the hash table after it has been completely initialized.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes an error in the failure path of leave_pinned. When
the rcache tries to enable leave_pinned but leave_pinned was not
specifically requested (opal_leave_pinned == -1) the code was
erroneously printing an error and returning NULL.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The opal_mem_hooks_release_hook does not have const on the pointer
(though it probably should). This commit eliminates a warning by
casting away the const until opal_mem_hooks_release_hook is updated to
use const.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This is complicated stuff: add some comments so that future
maintainers have some rationale to understand the way things have been
done.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Define OPAL_MAXHOSTNAMELEN to be either:
(MAXHOSTNAMELEN + 1) or
(limits.h:HOST_NAME_MAX + 1) or
(255 + 1)
For pmix code, define above using PMIX_MAXHOSTNAMELEN.
Fixup opal layer to use the new max.
Signed-off-by: Karol Mroz <mroz.karol@gmail.com>
The math for checking the number of QPs and CQs per usNIC/VF was
incorrect, allowing you to run MPI processes even when usNICs (i.e.,
VIC VFs) had fewer QPs and CQs than were necessary. This led to a
confusing error later when fi_enable(3) failed (because we lazily
create QPs). Fixing the math here ensure that we actually print a
helpful error message telling the user specifically what is wrong.
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
The heterogeneous code need to gracefully handly the contiguous
datatype loops in order to have the "#if 0" code path enabled again.
This is a performance issue (the correctness is guaranteed by the
current code).
These macros should really be named OPAL_SUMMARY_*; they're used in
all projects, and therefore should be in the lowest later project (OPAL).
Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
Fix CID 1358512: Error handling issues (NEGATIVE_RETURNS):
C libraries usually handle read (-1, ...) fine but it is safer to
avoid calling read with a negative handle. Added negative file
descriptor check.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Because of the removal of the linux memory component it is no longer
necessary to initialize the memory component in opal_init(). This
commit moves the initialization to the creation of the first rcache
component.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes bugs that can cause crashes and memory corruption
when the mremap hook is called. The problem occurs because of the
ellipses (...) in the mremap intercept function. The ellipses cover
the optional new_addr argument on Linux. This commit removes the
ellipses and adds an explicit 5th argument.
This commit also adds a hook for shmdt. The code only works on Linux
at the moment as it needs to read /proc/self/maps to determine the
size of the shared memory segment.
Additionally, this commit removes the mmap hook. There is no
apparent benefit for detecting mmap(..., PROT_NONE, ...) and it
seems to cause problems when threads are in use.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit removes the ptmalloc2 memory hooks. This is necessary in
order to support lazy registration of memory hooks. A feature that is
not supported by the ptmalloc hooks but is supported by the new
patcher hooks.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds a framework to abstract runtime code patching.
Components in the new framework can provide functions for either
patching a named function or a function pointer. The later
functionality is not being used but may provide a way to allow memory
hooks when dlopen functionality is disabled.
This commit adds two different flavors of code patching. The first is
provided by the overwrite component. This component overwrites the
first several instructions of the target function with code to jump to
the provided hook function. The hook is expected to provide the full
functionality of the hooked function.
The linux patcher component is based on the memory hooks in ucx. It
only works on linux and operates by overwriting function pointers in
the symbol table. In this case the hook is free to call the original
function using the function pointer returned by dlsym.
Both components restore the original functions when the patcher
framework closes.
Changes had to be made to support Power/PowerPC with the Linux
dynamic loader patcher. Some of the changes:
- Move code necessary for powerpc/power support to the patcher
base. The code is needed by both the overwrite and linux
components.
- Move patch structure down to base and move the patch list to
mca_patcher_base_module_t. The structure has been modified to
include a function pointer to the function that will unapply the
patch. This allows the mixing of multiple different types of
patches in the patch_list.
- Update linux patching code to keep track of the matching between
got entry and original (unpatched) address. This allows us to
completely clean up the patch on finalize.
All patchers keep track of the changes they made so that they can be
reversed when the patcher framework is closed.
At this time there are bugs in the Linux dynamic loader patcher so
its priority is lower than the overwrite patcher.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit makes it possible to set relative priorities for
components. Before the addition of the patched component there was
only one component that would run on any system but that is no longer
the case. When determining which component to open each component's
query function is called and the one that returns the highest priority
is opened. The default priority of the patcher component is set
slightly higher than the old ptmalloc2/ummunotify component.
This commit fixes a long-standing break in the abstration of the
memory components. ompi_mpi_init.c was referencing the linux malloc
hook initilize function to ensure the hooks are initialized for
libmpi.so. The abstraction break has been fixed by adding a memory
base function that calls the open memory component's malloc hook init
function if it has one. The code is not yet complete but is intended
to support ptmalloc in 2.0.0. In that case the base function will
always call the ptmalloc hook init if exists.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds support for runtime binary patching. The support is
broken down into two parts: util/opal_patcher.[ch] which contains the
functionality for runtime patching of symbols, and mca/memory/patcher
which patches the various symbols needed to provide support for memory
hooks. This work is preliminary and is based off work donated by IBM.
The patcher code is disabled if dlopen is disabled.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
https://github.com/pmix/master/pull/71
Have OMPI's current version of pmix120 nicely fail in case of
too long sun_path (longer than 108 or in case of OSX 103 chars).
And have OMPI return proper error messages with hints how to
amend.
Adding component name to the pvar pretty output as well.
Further, I think keeping the asprintf()/opal_info_out(msg,msg,------)
within the loops is needed to avoid printing any component information
(independently for group_vars and group_pvars) in case the upcoming
parameters are internal and not to be displayed.
Lastly, unnecessarily duplicating the dashed output should not happen as
each invocation of opal_info_show_mca_group_params() passes a new group
structure which we check.
Signed-off-by: Karol Mroz <mroz.karol@gmail.com>
When listing available parameters, add the component name to the MCA
framework field. Parsable option is already doing this, makes sense
for the pretty print option to do it as well.
Signed-off-by: Karol Mroz <mroz.karol@gmail.com>
When invoking, for example, `ompi_info` with:
-a
--params foo all
--params foo bar
it's useful to have the appropriate components and their versions be
displayed, regardless of whether they have registered any parameters.
Signed-off-by: Karol Mroz <mroz.karol@gmail.com>
This comit adds code that sets the fragment reg_cnt to 1 when sending
the completion message for an eager get. Without this the btl will
either hang or abort.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
