This commit removes the --with-mpi-thread-multiple option and forces
MPI_THREAD_MULTIPLE support. This cleans up an abstration violation
in opal where OMPI_ENABLE_THREAD_MULTIPLE determines whether the
opal_using_threads is meaningful. To reduce the performance hit on
MPI_THREAD_SINGLE programs an OPAL_UNLIKELY is used for the
check on opal_using_threads in OPAL_THREAD_* macros.
This commit does not clean up the arguments to the various functions
that take whether muti-threading support is enabled. That should be
done at a later time.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
1324740 - Resource leak
1304562 - Unchecked return value
1340514 - Dereference before null check
1340515 - Use of untrusted scalar value
1340516 - Use of untrusted string value
This commit allows to control output during abnormal oshmem/ompi application
termination.
Fixed issue in backtrace output. HAVE_BACKTRACE was never set so user was limited
in control of this variable.
Two related mca variables are moved to opal layer. Corresponding aliases are
added for ompi and oshmem.
Some MXM tls such as self, shm can comlete requests immediately.
Make sure that opal_progress() is called before before request
is completed.
fix ud_only logic when hw rdma channel is using ud and main
transport is rc or dc.
when profiling is built.
This prevents oshmem subroutines from being wrapped twice by third
party tools (e.g. once in oshmem and once in MPI)
see discussion starting at http://www.open-mpi.org/community/lists/devel/2015/08/17842.php
Thanks to Bert Wesarg for bringing this to our attention
ompi has new mpi_add_procs_cutoff argument that can control
creation of ompi_proc_t but We should be confident that all
ompi_proc_t object exists during oshmem_group_all creation.
Probably it could be done in more flexible way later.
Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>
Most functionality of oshmem_proc duplicates ompi_proc. In addition
to that, Current logic does not allow to do oshmem initialization
w/o ompi startup.
So this refactoring allows to avoid code duplication, decrease used
memory and make oshmem support easier.
Now oshmem_proc is transparent ompi_proc structure, that can be
extended by oshmem specific data.
Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>
Any buffer given to opal_dss.load becomes the responsibility of the
opal_buffer_t object. It will be freed automatically if either the
opal_buffer_t is released or opal_dss.load is called again on the
buffer. opal_dss.unload will not prevent this unless no unpacking
takes place between the .load and .unload calls.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Bring Slurm PMI-1 component online
Bring the s2 component online
Little cleanup - let the various PMIx modules set the process name during init, and then just raise it up to the ORTE level. Required as the different PMI environments all pass the jobid in different ways.
Bring the OMPI pubsub/pmi component online
Get comm_spawn working again
Ensure we always provide a cpuset, even if it is NULL
pmix/cray: adjust cray pmix component for pmix
Make changes so cray pmix can work within the integrated
ompi/pmix framework.
Bring singletons back online. Implement the comm_spawn operation using pmix - not tested yet
Cleanup comm_spawn - procs now starting, error in connect_accept
Complete integration
These man pages to OpenSHMEM API are received from openshmem.org
and ported with minimum modifications.
Thanks Tony Curtis for sharing man files.
Signed-off-by: Igor Ivanov <Igor.Ivanov@itseez.com>
* do not add -I/.../include/fca -I /.../include/fca_core to CPPFLAGS
* allow configure --with-fca
* search fca libs in both DIR/lib and DIR/lib64
* fix the description of the --with-fca option
only define the unique fortran symbol depending on
- CAPS
- PLAIN
- SINGLE_UNDERSCORE
- DOUBLE_UNDERSCORE
and bind the f08 symbol to the uniquely defined C symbol.
Use real data structures to make the code simpler.
(perl script written by Jeff)
This commit does two things. It removes checks for C99 required
headers (stdlib.h, string.h, signal.h, etc). Additionally it removes
definitions for required C99 types (intptr_t, int64_t, int32_t, etc).
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
Turns out that this is just copy-n-pasted code from OMPI. To be
clear: there's no need for the oshmem layer to instantiate sentinels
like mpi_fortran_bottom.
Thanks @jsquyres for pointing this.
This commit adds support for project_framework_component_* parameter
matching. This is the first step in allowing the same framework name
in multiple projects. This change also bumps the MCA component version
to 2.1.0.
All master frameworks have been updated to use the new component
versioning macro. An mca.h has been added to each project to add a
project specific versioning macro of the form
PROJECT_MCA_VERSION_2_1_0.
Signed-off-by: Nathan Hjelm <hjelmn@me.com>
Noe that this commit removes option:lt_dladvise from the various
"info" tools output. This technically breaks our CLI "ABI" because
we're not deprecating it / replacing it with an alias to some other
"into" tool output.
Although the dl/libltdl component contains an "have_lt_dladvise" MCA
var that contains the same information, the "option:lt_dladvise"
output from the various "info" tools is *not* an MCA var, and
therefore we can't alias it. So it just has to die.
In order to have an effect, ibv_fork_init should be called in the
beginning of the verbs initialization flow - before the calls to the
ibv_create_qp and ibv_create_cq verbs.
These functions are called from the oob/ud code and by the time the
other verbs components (btl openib, pml yalla, ...) call ibv_fork_init,
it's too late. This commit forces the call to ibv_fork_init (if it's
requested) right at the beginning of all the components that are using
verbs.
(ibv_fork_init() can be safely called multiple times)
This commit also removes the btl_openib_want_fork_support mca parameter
and adds a new mca parameter instead - opal_verbs_want_fork_support.
Through this new parameter, fork support may be requested for ALL
components.
The default value for this parameter is set to 1.
Before this commit the btl_openib_want_fork_support parameter didn't
provide fork support for the openib btl if its value was set to 1.
(because when openib called ibv_fork_init, it was already after the
calls to ibv_create_* in oob/ud and thereofre it failed).
Use of the old ompi_free_list_t and ompi_free_list_item_t is
deprecated. These classes will be removed in a future commit.
This commit updates the entire code base to use opal_free_list_t and
opal_free_list_item_t.
Notes:
OMPI_FREE_LIST_*_MT -> opal_free_list_* (uses opal_using_threads ())
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
The previous condition just called assert(), which is a no-op in
non-debug builds. Change it to print a message and then call abort()
to really actually above.
This was CID 1270155.
This commit make spml/yoda compatible with BTL 3.0. This is meant as a
starting point only. More work will be needed to make optimial use of
the new interface.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Coverity identified that we treated the possibility that one of the
message buffers could be NULL in some places (because strdup() could
fail), but not in others.
So just use stack buffers that will never be NULL.
This was CID 1269914.
Compiler warning that ret was used before it was assigned. Since this
part of the code is not performance-critical, just throw in an extra
assignment and be done with it.
send reply before posting the receive request again to limit the recursion size to
number of receive requests.
send can call opal_progress which calls this function again. If recv req is started
stack size will be proportional to number of job ranks.
We recognize that this means other users of OPAL will need to "wrap" the opal_process_name_t if they desire to abstract it in some fashion. This is regrettable, and we are looking at possible alternatives that might mitigate that requirement. Meantime, however, we have to put the needs of the OMPI community first, and are taking this step to restore hetero and SPARC support.
These two macros set the prefix for the OPAL and ORTE libraries,
respectively. Specifically, the OPAL library will be named
libPREFIXopen-pal.la and the ORTE library will be named
libPREFIXopen-rte.la.
These macros must be called, even if the prefix argument is empty.
The intent is that Open MPI will call these macros with an empty
prefix, but other projects (such as ORCM) will call these macros with
a non-empty prefix. For example, ORCM libraries can be named
liborcm-open-pal.la and liborcm-open-rte.la.
This scheme is necessary to allow running Open MPI applications under
systems that use their own versions of ORTE and OPAL. For example,
when running MPI applications under ORTE, if the ORTE and OPAL
libraries between OMPI and ORCM are not identical (which, because they
are released at different times, are likely to be different), we need
to ensure that the OMPI applications link against their ORTE and OPAL
libraries, but the ORCM executables link against their ORTE and OPAL
libraries.
the OPAL and ORTE libraries. This is required by projects such as ORCM
that have their own ORTE and OPAL libraries in order to avoid library
confusion. By renaming their version of the libraries, the OMPI
applications can correctly dynamically load the correct one for their
build."
This reverts commit 63f619f871.
Force completion of all puts before deregestering memheap/bss memory
Fixes a possible race condition where put request completion callback
is called when request context is already cleared.
Change-Id: I7ed887ec0b03a66ce5d3076a7edcf64061f57370
Do not allow combination of transports that is not compliant with
shmem spec. Especially do not allow mix of hw and software atomic
ops
Issue: 4721
Change-Id: Ide382f7510495df3d385f2a5ae5f9def6ef5332c
Use experimental verbs to allocate memory at fixed base
virtual address.
verbs will disqualify itself if shared_mr is disabled
or not supported and it is impossible to allocate memory
starting at fixed base virtual address.
verbs contig pages allocator did not guarantee fixed va, now it does.
(cherry picked from commit fd77ebd452)
Apply Jeff`s comments
Update with Jeff commits
(cherry picked from commit open-mpi/ompi-release@4dc487fc3d)
It looks like using MAP_PRIVATE instead of MAP_SHARED greatly
speeds up infiniband memory registration.
Change-Id: Id7089f58458ef8fff4034a2c4707d31f7e8b6694
Use experimental verbs to allocate memory at fixed base
virtual address.
verbs will disqualify itself if shared_mr is disabled
or not supported and it is impossible to allocate memory
starting at fixed base virtual address.
verbs contig pages allocator did not guarantee fixed va, now it does.
Added use_hp flag in sshmem/sysv variable to control huge page usage;
Added shared_mr sshmem/verbs;
Both paraemetes are set in auto.
Fix help messages
fixed by Igor, reviewed by @miked-mellanox and @alex-mikheev
Lock server pe computation was incorrect in cases when:
lock virtual address is signed long. In this case negative pe
value was returned.
In case when lock has different virtual adresses on different pes.
It can happen when memheap or static segment have different base
addresses. Use offset instead of absolute virtual address to
compute server pe
Fixed by @alex-mikheev, reviewed by @miked-mellanox
Update the VERSION file scheme:
* Remove "want_repo_rev".
* Add "tarball_version".
All values are now always included (major, minor, release, greek,
repo_rev). However, configure.ac now runs "opal_get_version.sh
... --tarball", which will return the value of tarball_version (if it
is non-empty) or the "full" version string (i.e.,
"major.minor.releasegreek").
WHAT: Open our low-level communication infrastructure by moving all necessary components (btl/rcache/allocator/mpool) down in OPAL
All the components required for inter-process communications are currently deeply integrated in the OMPI layer. Several groups/institutions have express interest in having a more generic communication infrastructure, without all the OMPI layer dependencies. This communication layer should be made available at a different software level, available to all layers in the Open MPI software stack. As an example, our ORTE layer could replace the current OOB and instead use the BTL directly, gaining access to more reactive network interfaces than TCP. Similarly, external software libraries could take advantage of our highly optimized AM (active message) communication layer for their own purpose. UTK with support from Sandia, developped a version of Open MPI where the entire communication infrastucture has been moved down to OPAL (btl/rcache/allocator/mpool). Most of the moved components have been updated to match the new schema, with few exceptions (mainly BTLs where I have no way of compiling/testing them). Thus, the completion of this RFC is tied to being able to completing this move for all BTLs. For this we need help from the rest of the Open MPI community, especially those supporting some of the BTLs. A non-exhaustive list of BTLs that qualify here is: mx, portals4, scif, udapl, ugni, usnic.
This commit was SVN r32317.
Update the comments to be correct.
cmr=v1.8.2:ticket=trac:4808
This commit was SVN r32300.
The following Trac tickets were found above:
Ticket 4808 --> https://svn.open-mpi.org/trac/ompi/ticket/4808
wrapper-data incorrectly adds @OMPI_WRAPPER_EXTRA_LIBS@. This fixes trac:4796 and should be
added to:
cmr=v1.8.2:reviewer=jsquyres
This commit was SVN r32299.
The following Trac tickets were found above:
Ticket 4796 --> https://svn.open-mpi.org/trac/ompi/ticket/4796
OpenSHMEMspec 1.1 introduces a set of environment variables that allows users to configure the Open-SHMEM implementation, and receive information about the implementation.
- Add SMA_SYMMETRIC_SIZE - number of bytes to allocate for symmetric heap
- SHMEM_SYMMETRIC_HEAP_SIZE (Mellanox extension) is used by a user to provide a size of symmetric area. This change sets this env variable in case a user does not set this variable
directly.
fixed by Igor, reviewed by Miked
cmr=v1.8.2:reviwer=ompi-rm1.8
This commit was SVN r32257.
mca_btl_base_segment_t and replace them with des_local and des_remote
This change also updates the BTL version to 3.0.0. This commit does
not represent the final version of BTL 3.0.0. More changes are coming.
In making this change I updated all of the BTLs as well as BTL user's
to use the new structure members. Please evaluate your component to
ensure the changes are correct.
RFC text:
This is the first of several BTL interface changes I am proposing for
the 1.9/2.0 release series.
What: Change naming of btl descriptor members. I propose we change
des_src and des_dst (and their associated counts) to be des_local and
des_remote. For receive callbacks the des_local member will be used to
communicate the segment information to the callback. The proposed change
will include updating all of the doxygen in btl.h as well as updating
all BTLs and BTL users to use the new naming scheme.
Why: My btl usage makes use of both put and get operations on the same
descriptor. With the current naming scheme I need to ensure that there
is consistency beteen the segments described in des_src and des_dst
depending on whether a put or get operation is executed. Additionally,
the current naming prevents BTLs that do not require prepare/RMA matched
operations (do not set MCA_BTL_FLAGS_RDMA_MATCHED) from executing
multiple simultaneous put AND get operations. At the moment the
descriptor can only be used with one or the other. The naming change
makes it easier for BTL users to setup/modify descriptors for RMA
operations as the local segment and remote segment are always in the
same member field. The only issue I forsee with this change is that it
will require a little more work to move BTL fixes to the 1.8 release
series.
This commit was SVN r32196.
Memory registration is aborted on first failure.
Already registered memory is freed and
correct error code is returned.
Memory deregistration always suceeeds
reviewed by miked
cmr=v1.8.2:reviewer=ompi-rm1.8
This commit was SVN r32175.
Use exising fields of mkey struct to identify 'shared memory'
segments.
mkey.u.key is now always initialized to MAP_SEGMENT_SHM_INVALID instead
of 0
reviewed by Mike and Igor
cmr=v1.8.2:reviewer=ompi-rm1.8
This commit was SVN r32174.
this commit fixes error propagation in :
- mca_memheap_base_reg
- mca_memheap_base_dereg
cmr=v1.8.2:reviewer=amikheev:ticket=4747
This commit was SVN r32173.
The following Trac tickets were found above:
Ticket 4747 --> https://svn.open-mpi.org/trac/ompi/ticket/4747
if source memory could not be registered, then return NULL
some cleanup might be needed, please refer to the FIXME in the code
cmr=v1.8.2:reviewer=miked
This commit was SVN r32081.
check for possibility of heap2heap copy was incorrect
in case when shared heaps have different virtual
addresses on same host.
It seems that ibv_exp_reg_mr() on CIB cards may return
different VAs for heap on same node. On CX3 addresses are
the same.
reviewed by miked
cmr=v1.8.2:reviewer=ompi-rm1.8
This commit was SVN r31969.
Few components had wrong mca variables registration procedure
List of them:
- atomic basic and mxm
- spml yoda and ikrit
Two mca variables as runtime_api_verbose and runtime_lock_recursive change
names to oshmem_api_verbose and oshmem_lock_recursive otherwise they
were not shown by oshmem_info tool.
fixed by Igor, reviewed by Miked
cmr=v1.8.2:reviewer=ompi-rm1.8
This commit was SVN r31962.
Deadlock when using the shmem_collect32()/shmem_collect64() routines and any of the non-root PEs pass 0 as the number of elements.
Algorithm in _algorithm_central_collector() does use 0 as a special value, and thus does not break out of the loop.
fixed by IgorI, reviewed by MikeD
cmr=v1.8.2:reviewer=ompi-rm1.8
This commit was SVN r31814.
top_ompi_srcdir -> OMPI_TOP_SRCDIR
top_ompi_builddir -> OMPI_TOP_BUILDDIR
We also split the srcdir/builddir flags according to their local tree (e.g., OPAL_TOP_SRCDIR), and tied them all together in configure.ac. Renamed ompi_ignore and ompi_unignore to be opal_<foo> as these are agnostic markers.
Only thing left is ompilibdir being treated similar to what we dif for srcdir/builddir. Coming soon.
This commit was SVN r31678.
new OFED changed struct layout and static assignment caused segv.
detect struct new layout and use dynamic assignment
fixed by AlexM, reviewed by Miked
cmr=v1.8.2:reviewer=ompi-rm1.8
This commit was SVN r31502.
- add some missing APIs
- change signature od shmem_finalize to comply with SGI spec
fixed by Roman, reviewed by Igor/Miked
Refs trac:4520
This commit was SVN r31413.
The following Trac tickets were found above:
Ticket 4520 --> https://svn.open-mpi.org/trac/ompi/ticket/4520
Use only contig allocator if shared_mr() is not supported
Fixes ibv_exp_reg_mr() parameters passing.
Fixes incorrect handling of sshmem_verbs_hca_name option.
fixed by AlexM, reviewed by Mike
cmr=v1.8.1:reviewer=ompi-rm1.8
This commit was SVN r31379.
set default memheap start address to use one returned from memallocator
on 32bit system kernel may have issue because of this:
CONFIG_STRICT_DEVMEM kernel option which "prevents user space access to physical memory above 1MB"
fixed by Roman, reviewed by Igor/Miked
cmr=v1.8.1:reviewer=ompi-rm1.8
This commit was SVN r31331.
Also, since I put some of the macros for these silent/verbose rules up
in the top-level Makefile.man-page-rules file, I renamed it to
Makefile.ompi-rules.
I've had this sitting around for a while; now seems like as good a
time as any to commit it.
This commit was SVN r31271.
Provide users with right fail reason.
fixes trac:4433
This commit was SVN r31202.
The following Trac tickets were found above:
Ticket 4433 --> https://svn.open-mpi.org/trac/ompi/ticket/4433
Refs trac:4399
fixed by Roman, reviewed by Miked
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r31179.
The following Trac tickets were found above:
Ticket 4399 --> https://svn.open-mpi.org/trac/ompi/ticket/4399
* Show the help message for all errors, not just EINVAL
* Put the help message in the correct helpfile
* Fix grammar and spelling, and expand the help message
cmr=v1.7.5:ticket=trac:4431
This commit was SVN r31173.
The following SVN revision numbers were found above:
r31172 --> open-mpi/ompi@3e51d28b97
The following Trac tickets were found above:
Ticket 4431 --> https://svn.open-mpi.org/trac/ompi/ticket/4431
Using MAP_ANONYMOUS brings different interpretation of fd, offset arguuments of mmap()
Linux:
The mapping is not backed by any file; the fd and offset arguments are ignored.
Mac:
Map anonymous memory not associated with any specific file. The offset argument is
ignored. Mac OS X specific: the file descriptor used for creating MAP_ANON regions
can be used to pass some Mach VM flags, and can be specified as -1 if no such flags
are associated with the region
FreeBSD:
Map anonymous memory not associated with any specific file. The file
descriptor used for creating MAP_ANON must be -1. The offset argument must be 0.
fixed by Igor, reviewed by Mike
Refs trac:4399
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r31140.
The following Trac tickets were found above:
Ticket 4399 --> https://svn.open-mpi.org/trac/ompi/ticket/4399
The verbose level was initialized too early
group shmem mca params together
fixed by Roman, reviewed by Igor/Mike
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r31107.
* Fixed the AC_ARG_ENABLE for the oshmem option.
* Fixed the AM_CONDITIONALs values for the projects.
* Re-added (and slightly simplified) the use of PROJECT_OSHMEM in
various Makefile.am's to disable building OSHMEM stuff
Submitted by Jeff, reviewed and approved by Ralph.
cmr=v1.7.5:reviewer=ompi-gk1.7
This commit was SVN r31062.
NOTE: I transferred the oshmem-disabled-by-default from the 1.7 branch to the trunk to minimize future disruption if/when we change that option.
cmr=v1.8:reviewer=jsquyres
This commit was SVN r31006.
The code is not used anywhere, so let's remove it until there are some
man pages to install.
Submitted by Jeff Squyres, reviewed by Josh Ladd.
cmr=v1.7.5:reviewer=ompi-gk1.7:subject=remove oshmem dead man page code
This commit was SVN r31002.
fixed by Igor, reviewed by Miked
fixes trac:4359
This commit was SVN r30996.
The following Trac tickets were found above:
Ticket 4359 --> https://svn.open-mpi.org/trac/ompi/ticket/4359
fix situations where cluster nodes can have different btls
Fixed by Roman, reviewed by Igor, Mike
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30877.
dtypes support to oshmem scoll mpi. added comment to
oshmem scoll mpi component regarding casting size_t to int.
Fixed by Elena, Reviewed by Igor/Mike
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30856.
- similar to opal/shmem
- next step is some refactoring and merge into opal/shmem
Developed by Igor, reviewed by AlexM, MikeD
This commit fixes trac:4261.
This commit was SVN r30855.
The following Trac tickets were found above:
Ticket 4261 --> https://svn.open-mpi.org/trac/ompi/ticket/4261
oshmem collective can select MPI collectives modules as provider.
Developed by Elena, reviewed by Miked
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30808.
- fix mxm/tl selection logic
- do not require memory registration if mxm/ud was selected
fixed by Alex, reviewed by Miked
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30802.
fix: do not fail on blm allocation error, wait for some puts to complete and retry
fixed by Roman, reviewed by Mike/Alex
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30779.
- MXM uses libtool versioning scheme which is enough, no need additional in OMPI
reviewed by yossi
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30768.
1. fix in oshmem scoll component: basic algorithms should
call basic collectives since their implementation
incompatible with others (fca, hcoll).
2. Set OPAL_EVLOOP_ONCE flag ON for libevent in the case
of yoda smpl. Otherwise there is possible deadlock in
atomic_basic_lock call
fixed by Val, Igor, reviewed by Miked
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30762.
- remove old, unused code
- rename mca param for oshmem preconnect to match mpi naming scheme.
fixed by Alex, reviewed by Mike
cmr=v1.7.5:reviewer=ompi-rm1.7
This commit was SVN r30748.
Allow only limited number of coonections to have 'puts'
that do not require remote completion ack. That will
greatly improve performance of shmem_fence()/shmem_quiet()
and shmem_barrier() when there are many active connections.
fixed by Alex, reviewed by Miked
Refs trac:3763
This commit was SVN r30573.
The following Trac tickets were found above:
Ticket 3763 --> https://svn.open-mpi.org/trac/ompi/ticket/3763
Fixes mxm endpoint destruction and hung during
SHMEM finalization.
Add a barrier between spml del procs and finalization.
Not having it caused hungs because ikrit spml can not properly
disconnect if its peer already finizalized.
Refs trac:3763
This commit was SVN r30572.
The following Trac tickets were found above:
Ticket 3763 --> https://svn.open-mpi.org/trac/ompi/ticket/3763
