ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность

README: sync with v2.x

Просмотр исходного кода 
The README on master had grown very, very stale.  This commit copies
the README from the tip of the v2.x branch (from
https://github.com/open-mpi/ompi/pull/3119) and preserves a few minor
differences between master and the v2.x branch.

Signed-off-by: Jeff Squyres <jsquyres@cisco.com>

[skip ci]
bot:notest
Этот коммит содержится в:
Jeff Squyres 2017-03-07 18:03:54 -05:00
родитель 7240bee0e0
Коммит 3a6b297bd5
1 изменённых файлов: 209 добавлений и 202 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

409
README
Просмотреть файл

@ -1,7 +1,7 @@
Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
University Research and Technology
Corporation. All rights reserved.
Copyright (c) 2004-2015 The University of Tennessee and The University
Copyright (c) 2004-2007 The University of Tennessee and The University
of Tennessee Research Foundation. All rights
reserved.
Copyright (c) 2004-2008 High Performance Computing Center Stuttgart,
@ -17,6 +17,9 @@ Copyright (c) 2010 Oak Ridge National Labs. All rights reserved.
Copyright (c) 2011 University of Houston. All rights reserved.
Copyright (c) 2013-2015 Intel, Inc. All rights reserved
Copyright (c) 2015 NVIDIA Corporation. All rights reserved.
Copyright (c) 2017 Los Alamos National Security, LLC. All rights
reserved.
$COPYRIGHT$
Additional copyrights may follow
@ -59,7 +62,7 @@ Much, much more information is also available in the Open MPI FAQ:
===========================================================================
The following abbreviated list of release notes applies to this code
base as of this writing (April 2015):
base as of this writing (March 2017):
General notes
-------------
@ -67,8 +70,8 @@ General notes
- Open MPI now includes two public software layers: MPI and OpenSHMEM.
Throughout this document, references to Open MPI implicitly include
both of these layers. When distinction between these two layers is
necessary, we will reference them as the "MPI" and "OSHMEM" layers
respectively.
necessary, we will reference them as the "MPI" and "OpenSHMEM"
layers respectively.
- OpenSHMEM is a collaborative effort between academia, industry, and
the U.S. Government to create a specification for a standardized API
@ -78,19 +81,8 @@ General notes
http://openshmem.org/
This OpenSHMEM implementation is provided on an experimental basis;
it has been lightly tested and will only work in Linux environments.
Although this implementation attempts to be portable to multiple
different environments and networks, it is still new and will likely
experience growing pains typical of any new software package.
End-user feedback is greatly appreciated.
This implementation will currently most likely provide optimal
performance on Mellanox hardware and software stacks. Overall
performance is expected to improve as other network vendors and/or
institutions contribute platform specific optimizations.
See below for details on how to enable the OpenSHMEM implementation.
This OpenSHMEM implementation will only work in Linux environments
with a restricted set of supported networks.
- Open MPI includes support for a wide variety of supplemental
hardware and software package. When configuring Open MPI, you may
@ -133,6 +125,9 @@ General notes
Intel, and Portland (*)
- OS X (10.8, 10.9, 10.10, 10.11), 32 and 64 bit (x86_64), with
XCode and Absoft compilers (*)
- MacOS (10.12), 64 bit (x85_64) with XCode and Absoft compilers (*)
- OpenBSD. Requires configure options --enable-mca-no-build=patcher
and --disable-slopen with this release.
(*) Be sure to read the Compiler Notes, below.
@ -176,16 +171,17 @@ Compiler Notes
pgi-9 : 9.0-4 known GOOD
pgi-10: 10.0-0 known GOOD
pgi-11: NO known good version with --enable-debug
pgi-12: 12.10 known BAD due to C99 compliance issues, and 12.8
and 12.9 both known BAD with --enable-debug
pgi-13: 13.10 known GOOD
pgi-12: 12.10 known BAD with -m32, but known GOOD without -m32
(and 12.8 and 12.9 both known BAD with --enable-debug)
pgi-13: 13.9 known BAD with -m32, 13.10 known GOOD without -m32
pgi-15: 15.10 known BAD with -m32
- Similarly, there is a known Fortran PGI compiler issue with long
source directory path names that was resolved in 9.0-4 (9.0-3 is
known to be broken in this regard).
- IBM's xlf compilers: NO known good version that can build/link
the MPI f08 bindings or build/link the OSHMEM Fortran bindings.
the MPI f08 bindings or build/link the OpenSHMEM Fortran bindings.
- On NetBSD-6 (at least AMD64 and i386), and possibly on OpenBSD,
libtool misidentifies properties of f95/g95, leading to obscure
@ -196,9 +192,14 @@ Compiler Notes
f95/g95), or by disabling the Fortran MPI bindings with
--disable-mpi-fortran.
- On OpenBSD/i386, if you configure with
--enable-mca-no-build=patcher, you will also need to add
--disable-dlopen. Otherwise, odd crashes can occur
nondeterministically.
- Absoft 11.5.2 plus a service pack from September 2012 (which Absoft
says is available upon request), or a version later than 11.5.2
(e.g., 11.5.3), is required to compile the new Fortran mpi_f08
(e.g., 11.5.3), is required to compile the Fortran mpi_f08
module.
- Open MPI does not support the Sparc v8 CPU target. However,
@ -254,6 +255,9 @@ Compiler Notes
version of the Intel 12.1 Linux compiler suite, the problem will go
away.
- It has been reported that Pathscale 5.0.5 and 6.0.527 compilers
give an internal compiler error when trying to Open MPI.
- Early versions of the Portland Group 6.0 compiler have problems
creating the C++ MPI bindings as a shared library (e.g., v6.0-1).
Tests with later versions show that this has been fixed (e.g.,
@ -289,6 +293,9 @@ Compiler Notes
still using GCC 3.x). Contact Pathscale support if you continue to
have problems with Open MPI's C++ bindings.
Note the MPI C++ bindings have been deprecated by the MPI Forum and
may not be supported in future releases.
- Using the Absoft compiler to build the MPI Fortran bindings on Suse
9.3 is known to fail due to a Libtool compatibility issue.
@ -298,7 +305,7 @@ Compiler Notes
********************************************************************
********************************************************************
*** There is now only a single Fortran MPI wrapper compiler and a
*** single Fortran OSHMEM wrapper compiler: mpifort and oshfort,
*** single Fortran OpenSHMEM wrapper compiler: mpifort and oshfort,
*** respectively. mpif77 and mpif90 still exist, but they are
*** symbolic links to mpifort.
********************************************************************
@ -349,12 +356,12 @@ Compiler Notes
is provided, allowing mpi_f08 to be used in new subroutines in
legacy MPI applications.
Per the OSHMEM specification, there is only one Fortran OSHMEM binding
provided:
Per the OpenSHMEM specification, there is only one Fortran OpenSHMEM
binding provided:
- shmem.fh: All Fortran OpenSHMEM programs **should** include 'shmem.fh',
and Fortran OSHMEM programs that use constants defined by OpenSHMEM
**MUST** include 'shmem.fh'.
- shmem.fh: All Fortran OpenSHMEM programs **should** include
'shmem.fh', and Fortran OpenSHMEM programs that use constants
defined by OpenSHMEM **MUST** include 'shmem.fh'.
The following notes apply to the above-listed Fortran bindings:
@ -386,10 +393,9 @@ Compiler Notes
Similar to the mpif.h interface, MPI_SIZEOF is only supported on
Fortran compilers that support INTERFACE and ISO_FORTRAN_ENV.
- The mpi_f08 module is new and has been tested with the Intel
Fortran compiler and gfortran >= 4.9. Other modern Fortran
compilers may also work (but are, as yet, only lightly tested).
It is expected that this support will mature over time.
- The mpi_f08 module has been tested with the Intel Fortran compiler
and gfortran >= 4.9. Other modern Fortran compilers likely also
work.
Many older Fortran compilers do not provide enough modern Fortran
features to support the mpi_f08 module. For example, gfortran <
@ -436,11 +442,11 @@ General Run-Time Support Notes
MPI Functionality and Features
------------------------------
- All MPI-3 functionality is supported.
- Rank reordering support is available using the TreeMatch library. It
is activated for the graph and dist_graph topologies.
- All MPI-3 functionality is supported.
- When using MPI deprecated functions, some compilers will emit
warnings. For example:
@ -455,22 +461,37 @@ MPI Functionality and Features
deprecated_example.c:4: warning: 'MPI_Type_struct' is deprecated (declared at /opt/openmpi/include/mpi.h:1522)
shell$
- MPI_THREAD_MULTIPLE support is included, but is only lightly tested.
It likely does not work for thread-intensive applications. Note
that *only* the MPI point-to-point communication functions for the
BTL's listed here are considered thread safe. Other support
functions (e.g., MPI attributes) have not been certified as safe
when simultaneously used by multiple threads.
- tcp
- sm
- MPI_THREAD_MULTIPLE is supported with some exceptions. Note that
Open MPI must be configured with --enable-mpi-thread-multiple to get
this level of thread safety support.
The following PMLs support MPI_THREAD_MULTIPLE:
- cm (see list (1) of supported MTLs, below)
- ob1 (see list (2) of supported BTLs, below)
- ucx
- yalla
(1) The cm PML and the following MTLs support MPI_THREAD_MULTIPLE:
- MXM
- ofi (Libfabric)
- portals4
(2) The ob1 PML and the following BTLs support MPI_THREAD_MULTIPLE:
- openib (see exception below)
- self
- sm
- smcuda
- tcp
- ugni
- usnic
- vader (shared memory)
Note that Open MPI's thread support is in a fairly early stage; the
above devices may *work*, but the latency is likely to be fairly
high. Specifically, efforts so far have concentrated on
*correctness*, not *performance* (yet).
The openib BTL's RDMACM based connection setup mechanism is also not
thread safe. The default UDCM method should be used for
applications requiring MPI_THREAD_MULTIPLE support.
YMMV.
Currently, MPI File operations are not thread safe even if MPI is
initialized for MPI_THREAD_MULTIPLE support.
- MPI_REAL16 and MPI_COMPLEX32 are only supported on platforms where a
portable C datatype can be found that matches the Fortran type
@ -498,17 +519,17 @@ MPI Functionality and Features
This library is being offered as a "proof of concept" / convenience
from Open MPI. If there is interest, it is trivially easy to extend
it to printf for other MPI functions. Patches and/or suggestions
would be greatfully appreciated on the Open MPI developer's list.
it to printf for other MPI functions. Pull requests on github.com
would be greatly appreciated.
OSHMEM Functionality and Features
------------------------------
OpenSHMEM Functionality and Features
------------------------------------
- All OpenSHMEM-1.0 functionality is supported.
- All OpenSHMEM-1.3 functionality is supported.
MPI Collectives
-----------
---------------
- The "hierarch" coll component (i.e., an implementation of MPI
collective operations) attempts to discover network layers of
@ -574,25 +595,26 @@ MPI Collectives
collectives, copies the data to staging buffers if GPU buffers, then
calls underlying collectives to do the work.
OSHMEM Collectives
-----------
OpenSHMEM Collectives
---------------------
- The "fca" scoll component: the Mellanox Fabric Collective Accelerator
(FCA) is a solution for offloading collective operations from the
MPI process onto Mellanox QDR InfiniBand switch CPUs and HCAs.
- The "fca" scoll component: the Mellanox Fabric Collective
Accelerator (FCA) is a solution for offloading collective operations
from the MPI process onto Mellanox QDR InfiniBand switch CPUs and
HCAs.
- The "basic" scoll component: Reference implementation of all OSHMEM
collective operations.
- The "basic" scoll component: Reference implementation of all
OpenSHMEM collective operations.
Network Support
---------------
- There are three main MPI network models available: "ob1", "cm", and
"yalla". "ob1" uses BTL ("Byte Transfer Layer") components for each
supported network. "cm" uses MTL ("Matching Tranport Layer")
components for each supported network. "yalla" uses the Mellanox
MXM transport.
- There are four main MPI network models available: "ob1", "cm",
"yalla", and "ucx". "ob1" uses BTL ("Byte Transfer Layer")
components for each supported network. "cm" uses MTL ("Matching
Tranport Layer") components for each supported network. "yalla"
uses the Mellanox MXM transport. "ucx" uses the OpenUCX transport.
- "ob1" supports a variety of networks that can be used in
combination with each other:
@ -605,30 +627,31 @@ Network Support
- SMCUDA
- Cisco usNIC
- uGNI (Cray Gemini, Aries)
- vader (XPMEM, Linux CMA, Linux KNEM, and general shared memory)
- vader (XPMEM, Linux CMA, Linux KNEM, and copy-in/copy-out shared
memory)
- "cm" supports a smaller number of networks (and they cannot be
used together), but may provide better overall MPI performance:
- QLogic InfiniPath / Intel True Scale PSM
- Intel Omni-Path PSM2
- Mellanox MXM
- Portals4
- Intel True Scale PSM (QLogic InfiniPath)
- OpenFabrics Interfaces ("libfabric" tag matching)
- Portals 4
Open MPI will, by default, choose to use "cm" when one of the
above transports can be used. Otherwise, "ob1" will be used and
the corresponding BTLs will be selected. Users can force the use
of ob1 or cm if desired by setting the "pml" MCA parameter at
run-time:
above transports can be used, unless OpenUCX or MXM support is
detected, in which case the "ucx" or "yalla" PML will be used
by default. Otherwise, "ob1" will be used and the corresponding
BTLs will be selected. Users can force the use of ob1 or cm if
desired by setting the "pml" MCA parameter at run-time:
shell$ mpirun --mca pml ob1 ...
or
shell$ mpirun --mca pml cm ...
- Similarly, there are two OSHMEM network models available: "yoda",
and "ikrit". "yoda" also uses the BTL components for many supported
network. "ikrit" interfaces directly with Mellanox MXM.
- Similarly, there are two OpenSHMEM network models available: "yoda",
and "ikrit". "yoda" also uses the BTL components for supported
networks. "ikrit" interfaces directly with Mellanox MXM.
- "yoda" supports a variety of networks that can be used:
@ -636,12 +659,13 @@ Network Support
- Loopback (send-to-self)
- Shared memory
- TCP
- usNIC
- "ikrit" only supports Mellanox MXM.
- MXM is the Mellanox Messaging Accelerator library utilizing a full
range of IB transports to provide the following messaging services
to the upper level MPI/OSHMEM libraries:
to the upper level MPI/OpenSHMEM libraries:
- Usage of all available IB transports
- Native RDMA support
@ -652,7 +676,7 @@ Network Support
- The usnic BTL is support for Cisco's usNIC device ("userspace NIC")
on Cisco UCS servers with the Virtualized Interface Card (VIC).
Although the usNIC is accessed via the OpenFabrics Libfabric API
stack, this BTL is specific to the Cisco usNIC device.
stack, this BTL is specific to Cisco usNIC devices.
- uGNI is a Cray library for communicating over the Gemini and Aries
interconnects.
@ -703,9 +727,9 @@ Network Support
Open MPI Extensions
-------------------
- An MPI "extensions" framework has been added (but is not enabled by
default). See the "Open MPI API Extensions" section below for more
information on compiling and using MPI extensions.
- An MPI "extensions" framework is included in Open MPI, but is not
enabled by default. See the "Open MPI API Extensions" section below
for more information on compiling and using MPI extensions.
- The following extensions are included in this version of Open MPI:
@ -715,9 +739,10 @@ Open MPI Extensions
- cr: Provides routines to access to checkpoint restart routines.
See ompi/mpiext/cr/mpiext_cr_c.h for a listing of available
functions.
- cuda: When the library is compiled with CUDA-aware support, it provides
two things. First, a macro MPIX_CUDA_AWARE_SUPPORT. Secondly, the
function MPIX_Query_cuda_support that can be used to query for support.
- cuda: When the library is compiled with CUDA-aware support, it
provides two things. First, a macro
MPIX_CUDA_AWARE_SUPPORT. Secondly, the function
MPIX_Query_cuda_support that can be used to query for support.
- example: A non-functional extension; its only purpose is to
provide an example for how to create other extensions.
@ -729,10 +754,9 @@ Building Open MPI
Open MPI uses a traditional configure script paired with "make" to
build. Typical installs can be of the pattern:
---------------------------------------------------------------------------
shell$ ./configure [...options...]
shell$ make all install
---------------------------------------------------------------------------
shell$ make [-j N] all install
(use an integer value of N for parallel builds)
There are many available configure options (see "./configure --help"
for a full list); a summary of the more commonly used ones is included
@ -755,16 +779,16 @@ INSTALLATION OPTIONS
files in <directory>/include, its libraries in <directory>/lib, etc.
--disable-shared
By default, libmpi and libshmem are built as a shared library, and
all components are built as dynamic shared objects (DSOs). This
switch disables this default; it is really only useful when used with
By default, Open MPI and OpenSHMEM build shared libraries, and all
components are built as dynamic shared objects (DSOs). This switch
disables this default; it is really only useful when used with
--enable-static. Specifically, this option does *not* imply
--enable-static; enabling static libraries and disabling shared
libraries are two independent options.
--enable-static
Build libmpi and libshmem as static libraries, and statically link in all
components. Note that this option does *not* imply
Build MPI and OpenSHMEM as static libraries, and statically link in
all components. Note that this option does *not* imply
--disable-shared; enabling static libraries and disabling shared
libraries are two independent options.
@ -785,14 +809,15 @@ INSTALLATION OPTIONS
is an important difference between the two:
"rpath": the location of the Open MPI libraries is hard-coded into
the MPI/OSHMEM application and cannot be overridden at run-time.
the MPI/OpenSHMEM application and cannot be overridden at
run-time.
"runpath": the location of the Open MPI libraries is hard-coded into
the MPI/OSHMEM application, but can be overridden at run-time by
setting the LD_LIBRARY_PATH environment variable.
the MPI/OpenSHMEM application, but can be overridden at run-time
by setting the LD_LIBRARY_PATH environment variable.
For example, consider that you install Open MPI vA.B.0 and
compile/link your MPI/OSHMEM application against it. Later, you install
Open MPI vA.B.1 to a different installation prefix (e.g.,
compile/link your MPI/OpenSHMEM application against it. Later, you
install Open MPI vA.B.1 to a different installation prefix (e.g.,
/opt/openmpi/A.B.1 vs. /opt/openmpi/A.B.0), and you leave the old
installation intact.
@ -849,7 +874,7 @@ NETWORKING SUPPORT / OPTIONS
Specify the directory where the Mellanox FCA library and
header files are located.
FCA is the support library for Mellanox QDR switches and HCAs.
FCA is the support library for Mellanox switches and HCAs.
--with-hcoll=<directory>
Specify the directory where the Mellanox hcoll library and header
@ -878,7 +903,8 @@ NETWORKING SUPPORT / OPTIONS
compiler/linker search paths.
Libfabric is the support library for OpenFabrics Interfaces-based
network adapters, such as Cisco usNIC, Intel True Scale PSM, etc.
network adapters, such as Cisco usNIC, Intel True Scale PSM, Cray
uGNI, etc.
--with-libfabric-libdir=<directory>
Look in directory for the libfabric libraries. By default, Open MPI
@ -938,7 +964,7 @@ NETWORKING SUPPORT / OPTIONS
if the PSM2 headers and libraries are not in default compiler/linker
search paths.
PSM2 is the support library for Intel Omni-Path network adapters.
PSM is the support library for Intel Omni-Path network adapters.
--with-psm2-libdir=<directory>
Look in directory for the PSM2 libraries. By default, Open MPI will
@ -950,13 +976,14 @@ NETWORKING SUPPORT / OPTIONS
Look in directory for Intel SCIF support libraries
--with-verbs=<directory>
Specify the directory where the verbs (also know as OpenFabrics, and
previously known as OpenIB) libraries and header files are located.
This option is generally only necessary if the verbs headers and
libraries are not in default compiler/linker search paths.
Specify the directory where the verbs (also known as OpenFabrics
verbs, or Linux verbs, and previously known as OpenIB) libraries and
header files are located. This option is generally only necessary
if the verbs headers and libraries are not in default
compiler/linker search paths.
"OpenFabrics" refers to operating system bypass networks, such as
InfiniBand, usNIC, iWARP, and RoCE (aka "IBoIP").
The Verbs library usually implies operating system bypass networks,
such as InfiniBand, usNIC, iWARP, and RoCE (aka "IBoIP").
--with-verbs-libdir=<directory>
Look in directory for the verbs libraries. By default, Open MPI
@ -992,9 +1019,6 @@ RUN-TIME SYSTEM SUPPORT
path names. --enable-orterun-prefix-by-default is a synonym for
this option.
--enable-sensors
Enable internal sensors (default: disabled).
--enable-orte-static-ports
Enable orte static ports for tcp oob (default: enabled).
@ -1202,15 +1226,9 @@ MPI FUNCTIONALITY
If --with-mpi-param is not specified, "runtime" is the default.
--enable-mpi-thread-multiple
Allows the MPI thread level MPI_THREAD_MULTIPLE.
This is currently disabled by default. Enabling
this feature will automatically --enable-opal-multi-threads.
--enable-opal-multi-threads
Enables thread lock support in the OPAL and ORTE layers. Does
not enable MPI_THREAD_MULTIPLE - see above option for that feature.
This is currently disabled by default.
--disable-mpi-thread-multiple
Disable the MPI thread level MPI_THREAD_MULTIPLE (it is enabled by
default).
--enable-mpi-cxx
Enable building the C++ MPI bindings (default: disabled).
@ -1245,7 +1263,7 @@ MPI FUNCTIONALITY
none: Synonym for "no".
no: Do not build any MPI Fortran support (same as
--disable-mpi-fortran). This is mutually exclusive
with building the OSHMEM Fortran interface.
with building the OpenSHMEM Fortran interface.
--enable-mpi-ext(=<list>)
Enable Open MPI's non-portable API extensions. If no <list> is
@ -1254,10 +1272,11 @@ MPI FUNCTIONALITY
See "Open MPI API Extensions", below, for more details.
--disable-mpi-io
Disable built-in support for MPI-2 I/O, likely because an externally-provided
MPI I/O package will be used. Default is to use the internal framework
system that uses the ompio component and a specially modified version of ROMIO
that fits inside the romio component
Disable built-in support for MPI-2 I/O, likely because an
externally-provided MPI I/O package will be used. Default is to use
the internal framework system that uses the ompio component and a
specially modified version of ROMIO that fits inside the romio
component
--disable-io-romio
Disable the ROMIO MPI-IO component
@ -1276,14 +1295,14 @@ MPI FUNCTIONALITY
significantly especially if you are creating large
communicators. (Disabled by default)
OSHMEM FUNCTIONALITY
OPENSHMEM FUNCTIONALITY
--disable-oshmem
Disable building the OpenSHMEM implementation (by default, it is
enabled).
--disable-oshmem-fortran
Disable building only the Fortran OSHMEM bindings. Please see
Disable building only the Fortran OpenSHMEM bindings. Please see
the "Compiler Notes" section herein which contains further
details on known issues with various Fortran compilers.
@ -1444,22 +1463,23 @@ NOTE: The version numbering conventions were changed with the release
Backwards Compatibility
-----------------------
Open MPI version vY is backwards compatible with Open MPI version vX
Open MPI version Y is backwards compatible with Open MPI version X
(where Y>X) if users can:
* Users can compile a correct MPI / OSHMEM program with vX
* Run it with the same CLI options and MCA parameters using vX or vY
* The job executes correctly
* Compile an MPI/OpenSHMEM application with version X, mpirun/oshrun
it with version Y, and get the same user-observable behavior.
* Invoke ompi_info with the same CLI options in versions X and Y and
get the same user-observable behavior.
Note that this definition encompasses several things:
* Application Binary Interface (ABI)
* MPI / OSHMEM run time system
* MPI / OpenSHMEM run time system
* mpirun / oshrun command line options
* MCA parameter names / values / meanings
However, this definition only applies when the same version of Open
MPI is used with all instances of the runtime and MPI / OSHMEM
MPI is used with all instances of the runtime and MPI / OpenSHMEM
processes in a single MPI job. If the versions are not exactly the
same everywhere, Open MPI is not guaranteed to work properly in any
scenario.
@ -1528,25 +1548,14 @@ The "A.B.C" version number may optionally be followed by a Quantifier:
Nightly development snapshot tarballs use a different version number
scheme; they contain three distinct values:
* The most recent Git tag name on the branch from which the tarball
was created.
* An integer indicating how many Git commits have occurred since
that Git tag.
* The Git hash of the tip of the branch.
* The git branch name from which the tarball was created.
* The date/timestamp, in YYYYMMDDHHMM format.
* The hash of the git commit from which the tarball was created.
For example, a snapshot tarball filename of
"openmpi-v1.8.2-57-gb9f1fd9.tar.bz2" indicates that this tarball was
created from the v1.8 branch, 57 Git commits after the "v1.8.2" tag,
specifically at Git hash gb9f1fd9.
Open MPI's Git master branch contains a single "dev" tag. For
example, "openmpi-dev-8-gf21c349.tar.bz2" represents a snapshot
tarball created from the master branch, 8 Git commits after the "dev"
tag, specifically at Git hash gf21c349.
The exact value of the "number of Git commits past a tag" integer is
fairly meaningless; its sole purpose is to provide an easy,
human-recognizable ordering for snapshot tarballs.
"openmpi-v2.x-201703070235-e4798fb.tar.gz" indicates that this tarball
was created from the v2.x branch, on March 7, 2017, at 2:35am GMT,
from git hash e4798fb.
Shared Library Version Number
-----------------------------
@ -1596,11 +1605,11 @@ Here's how we apply those rules specifically to Open MPI:
above rules: rules 4, 5, and 6 only apply to the official MPI and
OpenSHMEM interfaces (functions, global variables). The rationale
for this decision is that the vast majority of our users only care
about the official/public MPI/OSHMEM interfaces; we therefore want
the .so version number to reflect only changes to the official
MPI/OSHMEM APIs. Put simply: non-MPI/OSHMEM API / internal
changes to the MPI-application-facing libraries are irrelevant to
pure MPI/OSHMEM applications.
about the official/public MPI/OpenSHMEM interfaces; we therefore
want the .so version number to reflect only changes to the
official MPI/OpenSHMEM APIs. Put simply: non-MPI/OpenSHMEM API /
internal changes to the MPI-application-facing libraries are
irrelevant to pure MPI/OpenSHMEM applications.
* libmpi
* libmpi_mpifh
@ -1667,15 +1676,16 @@ tests:
receives a few MPI messages (e.g., the ring_c program in the
examples/ directory in the Open MPI distribution).
4. Use "oshrun" to launch a non-OSHMEM program across multiple nodes.
4. Use "oshrun" to launch a non-OpenSHMEM program across multiple
nodes.
5. Use "oshrun" to launch a trivial MPI program that does no OSHMEM
communication (e.g., hello_shmem.c program in the examples/ directory
in the Open MPI distribution.)
5. Use "oshrun" to launch a trivial MPI program that does no OpenSHMEM
communication (e.g., hello_shmem.c program in the examples/
directory in the Open MPI distribution.)
6. Use "oshrun" to launch a trivial OSHMEM program that puts and gets
a few messages. (e.g., the ring_shmem.c in the examples/ directory
in the Open MPI distribution.)
6. Use "oshrun" to launch a trivial OpenSHMEM program that puts and
gets a few messages. (e.g., the ring_shmem.c in the examples/
directory in the Open MPI distribution.)
If you can run all six of these tests successfully, that is a good
indication that Open MPI built and installed properly.
@ -1751,7 +1761,7 @@ Compiling Open MPI Applications
-------------------------------
Open MPI provides "wrapper" compilers that should be used for
compiling MPI and OSHMEM applications:
compiling MPI and OpenSHMEM applications:
C: mpicc, oshcc
C++: mpiCC, oshCC (or mpic++ if your filesystem is case-insensitive)
@ -1762,7 +1772,7 @@ For example:
shell$ mpicc hello_world_mpi.c -o hello_world_mpi -g
shell$
For OSHMEM applications:
For OpenSHMEM applications:
shell$ oshcc hello_shmem.c -o hello_shmem -g
shell$
@ -1862,17 +1872,18 @@ Note that the values of component parameters can be changed on the
mpirun / mpiexec command line. This is explained in the section
below, "The Modular Component Architecture (MCA)".
Open MPI supports oshrun to launch OSHMEM applications. For example:
Open MPI supports oshrun to launch OpenSHMEM applications. For
example:
shell$ oshrun -np 2 hello_world_oshmem
OSHMEM applications may also be launched directly by resource managers
such as SLURM. For example, when OMPI is configured --with-pmi and
--with-slurm one may launch OSHMEM applications via srun:
OpenSHMEM applications may also be launched directly by resource
managers such as SLURM. For example, when OMPI is configured
--with-pmi and --with-slurm, one may launch OpenSHMEM applications via
srun:
shell$ srun -N 2 hello_world_oshmem
===========================================================================
The Modular Component Architecture (MCA)
@ -1886,10 +1897,8 @@ component frameworks in Open MPI:
MPI component frameworks:
-------------------------
bcol - Base collective operations
bml - BTL management layer
coll - MPI collective algorithms
crcp - Checkpoint/restart coordination protocol
fbtl - file byte transfer layer: abstraction for individual
read/write operations for OMPIO
fcoll - collective read and write operations for MPI I/O
@ -1901,21 +1910,20 @@ op - Back end computations for intrinsic MPI_Op operators
osc - MPI one-sided communications
pml - MPI point-to-point management layer
rte - Run-time environment operations
sbgp - Collective operation sub-group
sharedfp - shared file pointer operations for MPI I/O
topo - MPI topology routines
vprotocol - Protocols for the "v" PML
OSHMEM component frameworks:
OpenSHMEM component frameworks:
-------------------------
atomic - OSHMEM atomic operations
memheap - OSHMEM memory allocators that support the
atomic - OpenSHMEM atomic operations
memheap - OpenSHMEM memory allocators that support the
PGAS memory model
scoll - OSHMEM collective operations
spml - OSHMEM "pml-like" layer: supports one-sided,
scoll - OpenSHMEM collective operations
spml - OpenSHMEM "pml-like" layer: supports one-sided,
point-to-point operations
sshmem - OSHMEM shared memory backing facility
sshmem - OpenSHMEM shared memory backing facility
Back-end run-time environment (RTE) component frameworks:
@ -1937,8 +1945,6 @@ rml - RTE message layer
routed - Routing table for the RML
rtc - Run-time control framework
schizo - OpenRTE personality framework
snapc - Snapshot coordination
sstore - Distributed scalable storage
state - RTE state machine
Miscellaneous frameworks:
@ -1946,9 +1952,7 @@ Miscellaneous frameworks:
allocator - Memory allocator
backtrace - Debugging call stack backtrace support
btl - point-to-point Byte Transfer Layer
compress - Compression algorithms
crs - Checkpoint and restart service
btl - Point-to-point Byte Transfer Layer
dl - Dynamic loading library interface
event - Event library (libevent) versioning support
hwloc - Hardware locality (hwloc) versioning support
@ -1962,9 +1966,8 @@ patcher - Symbol patcher hooks
pmix - Process management interface (exascale)
pstat - Process status
rcache - Memory registration cache
reachable - Network reachability computations
sec - Security framework
shmem - Shared memory support (NOT related to OSHMEM)
shmem - Shared memory support (NOT related to OpenSHMEM)
timer - High-resolution timers
---------------------------------------------------------------------------
@ -1981,8 +1984,8 @@ to see what its tunable parameters are. For example:
shell$ ompi_info --param btl tcp
shows a some of parameters (and default values) for the tcp btl
component.
shows some of the parameters (and default values) for the tcp btl
component (use --level to show *all* the parameters; see below).
Note that ompi_info only shows a small number a component's MCA
parameters by default. Each MCA parameter has a "level" value from 1
@ -1997,18 +2000,18 @@ MPI, we have interpreted these nine levels as three groups of three:
5. Application tuner / detailed
6. Application tuner / all
7. MPI/OSHMEM developer / basic
8. MPI/OSHMEM developer / detailed
9. MPI/OSHMEM developer / all
7. MPI/OpenSHMEM developer / basic
8. MPI/OpenSHMEM developer / detailed
9. MPI/OpenSHMEM developer / all
Here's how the three sub-groups are defined:
1. End user: Generally, these are parameters that are required for
correctness, meaning that someone may need to set these just to
get their MPI/OSHMEM application to run correctly.
get their MPI/OpenSHMEM application to run correctly.
2. Application tuner: Generally, these are parameters that can be
used to tweak MPI application performance.
3. MPI/OSHMEM developer: Parameters that either don't fit in the
3. MPI/OpenSHMEM developer: Parameters that either don't fit in the
other two, or are specifically intended for debugging /
development of Open MPI itself.
@ -2065,10 +2068,10 @@ variable; an environment variable will override the system-wide
defaults.
Each component typically activates itself when relevant. For example,
the MX component will detect that MX devices are present and will
automatically be used for MPI communications. The SLURM component
will automatically detect when running inside a SLURM job and activate
itself. And so on.
the usNIC component will detect that usNIC devices are present and
will automatically be used for MPI communications. The SLURM
component will automatically detect when running inside a SLURM job
and activate itself. And so on.
Components can be manually activated or deactivated if necessary, of
course. The most common components that are manually activated,
@ -2082,10 +2085,14 @@ comma-delimited list to the "btl" MCA parameter:
shell$ mpirun --mca btl tcp,self hello_world_mpi
To add shared memory support, add "sm" into the command-delimited list
(list order does not matter):
To add shared memory support, add "vader" into the command-delimited
list (list order does not matter):
shell$ mpirun --mca btl tcp,sm,self hello_world_mpi
shell$ mpirun --mca btl tcp,vader,self hello_world_mpi
(there is an "sm" shared memory BTL, too, but "vader" is a newer
generation of shared memory support; by default, "vader" will be used
instead of "sm")
To specifically deactivate a specific component, the comma-delimited
list can be prepended with a "^" to negate it:
@ -2130,10 +2137,10 @@ user's list:
http://lists.open-mpi.org/mailman/listinfo/users
Developer-level bug reports, questions, and comments should generally
be sent to the developer's mailing list (devel@lists.open-mpi.org). Please
do not post the same question to both lists. As with the user's list,
only subscribers are allowed to post to the developer's list. Visit
the following web page to subscribe:
be sent to the developer's mailing list (devel@lists.open-mpi.org).
Please do not post the same question to both lists. As with the
user's list, only subscribers are allowed to post to the developer's
list. Visit the following web page to subscribe:
http://lists.open-mpi.org/mailman/listinfo/devel