openmpi/doc/user/mpi-details.tex

% -*- latex -*-
%
% Copyright (c) 2004-2005 The Trustees of Indiana University.
%                         All rights reserved.
% Copyright (c) 2004-2005 The Trustees of the University of Tennessee.
%                         All rights reserved.
% Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, 
%                         University of Stuttgart.  All rights reserved.
% Copyright (c) 2004-2005 The Regents of the University of California.
%                         All rights reserved.
% $COPYRIGHT$
% 
% Additional copyrights may follow
% 
% $HEADER$
%

\chapter{Supported MPI Functionality}
\label{sec:mpi-functionality}

This chapter discusses the exact levels of MPI functionality that is
supported by Open MPI.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{MPI-1 Support}

Open MPI \ompiversion\ has support for all MPI-1 functionality.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Language Bindings}
\label{sec:mpi-details-mpi1-language}

Open MPI provides C, C++, Fortran 77, and Fortran 90 bindings for all
MPI-1 functions, types, and constants.
%
Profiling support is available in all four languages (if Open MPI was
configured and compiled with profiling support).  The \icmd{ompi\_info}
command can be used to see if profiling support was included in
Open MPI.

{\Huge JMS needs massive overhauling}

Support for optional Fortran types has now been added.
Table~\ref{tbl:mpi-details-optional-fortran-dtypes} lists the new
datatypes.  Note that \mpidatatype{MPI\_\-INTEGER8} and
\mpidatatype{MPI\_\-REAL16} are listed even though they are not
defined by the MPI standard.  Support for these types is included per
request from Open MPI users.

\begin{table}[htbp]
  \centering
  \begin{tabular}{|ll|}
    \hline
    \multicolumn{2}{|c|}{Supported Datatypes} \\
    \hline
    \hline
    \mpidatatype{MPI\_\-INTEGER1} &
    \mpidatatype{MPI\_\-INTEGER2} \\
%
    \mpidatatype{MPI\_\-INTEGER4} &
    \mpidatatype{MPI\_\-INTEGER8} \\
%
    \mpidatatype{MPI\_\-REAL4} &
    \mpidatatype{MPI\_\-REAL8} \\
%
    \mpidatatype{MPI\_\-REAL16} &
    ~ \\
    \hline
  \end{tabular}
  \caption{Supported optional fortran datatypes.}
  \label{tbl:mpi-details-optional-fortran-dtypes}
\end{table}

\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{\mpifunc{MPI\_\-CANCEL}}

\mpifunc{MPI\_\-CANCEL} works properly for receives, but will almost
never work on sends.  \mpifunc{MPI\_\-CANCEL} is most frequently used
with unmatched \mpifunc{MPI\_\-IRECV}'s that were made ``in case'' a
matching message arrived.  This simply entails removing the receive
request from the local queue, and is fairly straightforward to
implement.

Actually canceling a send operation is much more difficult because
some meta information about a message is usually sent immediately.  As
such, the message is usually at least partially sent before an
\mpifunc{MPI\_\-CANCEL} is issued.  Trying to chase down all the
particular cases is a nightmare, to say the least.

As such, the Open MPI Team decided not to implement \mpifunc{MPI\_\-CANCEL}
on sends, and instead concentrate on other features.

But in true MPI Forum tradition, we would be happy to discuss any code
that someone would like to submit that fully implements
\mpifunc{MPI\_\-CANCEL}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
\section{MPI-2 Support}

Open MPI \lamversion\ has support for many MPI-2 features.  The main
chapters of the MPI-2 standard are listed below, along with a summary
of the support provided for each chapter.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Miscellany}

%%%%%

\paragraph{Portable MPI Process Startup.}

The \icmd{mpiexec} command is now supported.  Common examples include:

\lstset{style=lam-shell}
\begin{lstlisting}
# Runs 4 copes of the MPI program my_mpi_program
shell$ mpiexec -n 4 my_mpi_program

# Runs my_linux_program on all available Linux machines, and runs
# my_solaris_program on all available Solaris machines
shell$ mpiexec -arch linux my_linux_program : -arch solaris my_solaris_program

# Boot the Open MPI run-time environment, run my_mpi_program on all
# available CPUs, and then shut down the Open MPI run-time environment.
shell$ mpiexec -machinefile hostfile my_mpi_program
\end{lstlisting}
% stupid emacs mode: $

See the \file{mpiexec(1)} man page for more details on supported
options as well as more examples.

%%%%%

\paragraph{Passing \const{NULL} to \mpifunc{MPI\_\-INIT}.}

Passing \const{NULL} as both arguments to \mpifunc{MPI\_\-INIT} is
fully supported.

%%%%%

\paragraph{Version Number.}

Open MPI \lamversion\ reports its MPI version as 1.2 through the function
\mpifunc{MPI\_\-GET\_\-VERSION}.

%%%%%

\paragraph{Datatype Constructor \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-INDEXED\_\-BLOCK}.}

The MPI function \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-INDEXED\_\-BLOCK}
is not supported by Open MPI.

%%%%%

\paragraph{Treatment of \mpitype{MPI\_\-Status}.}

Although Open MPI supports the constants \mpiconst{MPI\_\-STATUS\_\-IGNORE}
and \mpiconst{MPI\_\-STATUSES\_\-IGNORE}, the function
\mpifunc{MPI\_\-REQUEST\_\-GET\_\-STATUS} is not provided.

%%%%%

\paragraph{Error class for invalid keyval.}

The error class for invalid keyvals, \mpiconst{MPI\_\-ERR\_\-KEYVAL},
is fully supported.

%%%%%

\paragraph{Committing committed datatype.}

Committing a committed datatype is fully supported; its end effect is
a no-op.

%%%%%

\paragraph{Allowing user functions at process termination.}

Attaching attributes to \mpiconst{MPI\_\-COMM\_\-SELF} that have
user-specified delete functions will now trigger these functions to be
invoked as the first phase of \mpifunc{MPI\_\-FINALIZE}.  When these
functions are run, MPI is still otherwise fully functional.

%%%%%

\paragraph{Determining whether MPI has finished.}

The function \mpifunc{MPI\_\-FINALIZED} is fully supported.

%%%%%

\paragraph{The {\sf Info} object.}

Full support for \mpitype{MPI\_\-Info} objects is provided.  See
Table~\ref{tbl:mpi-details-info-functions}.

\begin{table}[htbp]
  \centering
  \begin{tabular}{|lll|}
    \hline
    \multicolumn{3}{|c|}{Supported Functions} \\
    \hline
    \hline
    \mpifunc{MPI\_\-INFO\_\-CREATE} &
    \mpifunc{MPI\_\-INFO\_\-FREE} &
    \mpifunc{MPI\_\-INFO\_\-GET\_\-NTHKEY} \\
%
    \mpifunc{MPI\_\-INFO\_\-DELETE} &
    \mpifunc{MPI\_\-INFO\_\-GET} & 
    \mpifunc{MPI\_\-INFO\_\-GET\_\-VALUELEN} \\
%
    \mpifunc{MPI\_\-INFO\_\-DUP} &
    \mpifunc{MPI\_\-INFO\_\-GET\_\-NKEYS} &
    \mpifunc{MPI\_\-INFO\_\-SET} \\
    \hline
  \end{tabular}
  \caption{Supported MPI-2 info functions.}
  \label{tbl:mpi-details-info-functions}
\end{table}

%%%%%

\paragraph{Memory allocation.}

The \mpifunc{MPI\_\-ALLOC\_\-MEM} and \mpifunc{MPI\_\-FREE\_\-MEM}
functions will return ``special'' memory that enable fast memory
passing in RPIs that support it.
%
These functions are simply wrappers to \func{malloc()} and
\func{free()} (respectively) in RPI modules that do not take advantage
of ``special'' memory.
%
These functions can be used portably for potential performance gains.

%%%%%

\paragraph{Language interoperability.}

Inter-language interoperability is supported.  It is possible to
initialize Open MPI from either C or Fortran and mix MPI calls from
both languages.  Handle conversions for inter-language
interoperability are fully supported.  See
Table~\ref{tbl:mpi-details-handle-conversion-functions}.

\begin{table}[htbp]
  \centering
  \begin{tabular}{|ll|}
    \hline
    \multicolumn{2}{|c|}{Supported Functions} \\
    \hline
    \hline
    \mpifunc{MPI\_\-COMM\_\-F2C} &
    \mpifunc{MPI\_\-COMM\_\-C2F} \\
%  
    \mpifunc{MPI\_\-GROUP\_\-F2C} &
    \mpifunc{MPI\_\-GROUP\_\-C2F} \\
%
    \mpifunc{MPI\_\-TYPE\_\-F2C} &
    \mpifunc{MPI\_\-TYPE\_\-C2F} \\
%  
    \mpifunc{MPI\_\-REQUEST\_\-F2C} &
    \mpifunc{MPI\_\-REQUEST\_\-C2F} \\
%  
    \mpifunc{MPI\_\-INFO\_\-F2C} &
    \mpifunc{MPI\_\-INFO\_\-C2F} \\
%  
    \mpifunc{MPI\_\-WIN\_\-F2C} &
    \mpifunc{MPI\_\-WIN\_\-C2F} \\
%  
    \mpifunc{MPI\_\-STATUS\_\-F2C} &
    \mpifunc{MPI\_\-STATUS\_\-C2F} \\
    \hline
  \end{tabular}
  \caption{Supported MPI-2 handle conversion functions.}
  \label{tbl:mpi-details-handle-conversion-functions}
\end{table}

%%%%%

\paragraph{Error handlers.}

Communicator and window error handler functions are fully supported;
this functionality is not yet supported for \mpitype{MPI\_\-File}
handles.  See Table~\ref{tbl:mpi-details-err-handler-functions}

\begin{table}[htbp]
  \centering
  \begin{tabular}{|ll|}
    \hline
    \multicolumn{2}{|c|}{Supported Functions} \\
    \hline
    \hline
    \mpifunc{MPI\_\-COMM\_\-CREATE\_\-ERRHANDLER} &
    \mpifunc{MPI\_\-WIN\_\-CREATE\_\-ERRHANDLER} \\
%
    \mpifunc{MPI\_\-COMM\_\-GET\_\-ERRHANDLER} &
    \mpifunc{MPI\_\-WIN\_\-GET\_\-ERRHANDLER} \\
%
    \mpifunc{MPI\_\-COMM\_\-SET\_\-ERRHANDLER} &
    \mpifunc{MPI\_\-WIN\_\-SET\_\-ERRHANDLER} \\
    \hline
  \end{tabular}
  \caption{Supported MPI-2 error handler functions.}
  \label{tbl:mpi-details-err-handler-functions}
\end{table}

%%%%%

\paragraph{New datatype manipulation functions.}

Several new datatype manipulation functions are provided.
Table~\ref{tbl:mpi-details-new-dtype-manip-functions} lists the new
functions.

\begin{table}[htbp]
  \centering
  \begin{tabular}{|ll|}
    \hline
    \multicolumn{2}{|c|}{Supported Functions} \\
    \hline
    \hline
    \mpifunc{MPI\_\-GET\_\-ADDRESS} &
    \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-SUBARRAY} \\
%
    \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-DARRAY} &
    \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-STRUCT} \\
%
    \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-HINDEXED} &
    \mpifunc{MPI\_\-TYPE\_\-GET\_\-EXTENT} \\
%
    \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-HVECTOR} &
    \mpifunc{MPI\_\-TYPE\_\-GET\_\-TRUE\_\-EXTENT} \\
%
    \mpifunc{MPI\_\-TYPE\_\-CREATE\_\-RESIZED} &
    ~ \\
    \hline
  \end{tabular}
  \caption{Supported MPI-2 new datatype manipulation functions.}
  \label{tbl:mpi-details-new-dtype-manip-functions}
\end{table}

%%%%%

\paragraph{New predefined datatypes.}

Support has been added for the
\mpidatatype{MPI\_\-LONG\_\-LONG\_\-INT},
\mpidatatype{MPI\_\-UNSIGNED\_\-LONG\_\-LONG} and
\mpidatatype{MPI\_\-WCHAR} basic datatypes.

%%%%%

\paragraph{Canonical \mpifunc{MPI\_\-PACK}, \mpifunc{MPI\_\-UNPACK}.}

Support is not provided for \mpifunc{MPI\_\-PACK\_\-EXTERNAL},
\mpifunc{MPI\_\-UNPACK\_\-EXTERNAL}, or
\mpifunc{MPI\_\-PACK\_\-EXTERNAL\_\-SIZE}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Process Creation and Management}

Open MPI supports all MPI-2 dynamic process management.
Table~\ref{tbl:mpi-details-dynamic-functions} lists all the supported
functions.

\begin{table}[htbp]
  \centering
  \begin{tabular}{|lll|}
    \hline
    \multicolumn{3}{|c|}{Supported Functions} \\
    \hline
    \hline
    \mpifunc{MPI\_\-CLOSE\_\-PORT} &
    \mpifunc{MPI\_\-COMM\_\-GET\_\-PARENT} &
    \mpifunc{MPI\_\-LOOKUP\_\-NAME} \\
%
    \mpifunc{MPI\_\-COMM\_\-ACCEPT} &
    \mpifunc{MPI\_\-COMM\_\-JOIN} &
    \mpifunc{MPI\_\-OPEN\_\-PORT} \\
%
    \mpifunc{MPI\_\-COMM\_\-SPAWN} &
    \mpifunc{MPI\_\-COMM\_\-CONNECT} &
    \mpifunc{MPI\_\-PUBLISH\_\-NAME} \\
%
    \mpifunc{MPI\_\-COMM\_\-DISCONNECT} &
    \mpifunc{MPI\_\-COMM\_\-SPAWN\_\-MULTIPLE} &
    \mpifunc{MPI\_\-UNPUBLISH\_\-NAME} \\
    \hline
  \end{tabular}
  \caption{Supported MPI-2 dynamic functions.}
  \label{tbl:mpi-details-dynamic-functions}
\end{table}

As requested by Open MPI users, \mpifunc{MPI\_\-COMM\_\-SPAWN} and
\mpifunc{MPI\_\-COMM\_\-SPAWN\_\-MULTIPLE} supports some
\mpitype{MPI\_\-Info} keys for spawning MPMD applications and for more
fine-grained control about where children processes are spawned.  See
the \file{MPI\_\-Comm\_\-spawn(3)} man page for more details.

These functions supersede the \mpifunc{MPIL\_\-COMM\_\-SPAWN} function
that Open MPI introduced in version 6.2b.  Hence,
\mpifunc{MPIL\_\-COMM\_\-SPAWN} is no longer available.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{One-Sided Communication}

Support is provided for get/put/accumulate data transfer operations
and for the post/wait/start/complete and fence synchronization
operations.  No support is provided for window locking.

The datatypes used in the get/put/accumulate operations are restricted
to being basic datatypes or single level contiguous/vectors of basic
datatypes.

The implementation of the one-sided operations is layered on top of
the point-to-point functions, and will thus perform no better than
them.  Nevertheless it is hoped that providing this support will aid
developers in developing and debugging codes using one-sided
communication.

Table~\ref{tbl:mpi-details-one-sided-functions} lists the functions
related to one-sided communication that have been implemented.

\begin{table}[htbp]
  \centering
  \begin{tabular}{|lll|}
    \hline
    \multicolumn{3}{|c|}{Supported Functions} \\
    \hline
    \hline
    \mpifunc{MPI\_\-ACCUMULATE} &
    \mpifunc{MPI\_\-WIN\_\-CREATE} &
    \mpifunc{MPI\_\-WIN\_\-POST} \\
%
    \mpifunc{MPI\_\-GET} &
    \mpifunc{MPI\_\-WIN\_\-FENCE} &
    \mpifunc{MPI\_\-WIN\_\-START} \\
%
    \mpifunc{MPI\_\-PUT} &
    \mpifunc{MPI\_\-WIN\_\-FREE} &
    \mpifunc{MPI\_\-WIN\_\-WAIT} \\
%
    \mpifunc{MPI\_\-WIN\_\-COMPLETE} &
    \mpifunc{MPI\_\-WIN\_\-GET\_\-GROUP} &
    ~ \\
    \hline
  \end{tabular}
  \caption{Supported MPI-2 one-sided functions.}
  \label{tbl:mpi-details-one-sided-functions}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Extended Collective Operations}
\label{sec:mpi:extended-collectives}

\changebegin{7.1}

Open MPI implements the new MPI-2 collective functions
\mpifunc{MPI\_\-EXSCAN} and \mpifunc{MPI\_\-ALLTOALLW} for
intracommunicators.

Intercommunicator collectives are implemented for all the functions
listed in Table~\ref{tbl:mpi-details-intercomm-collective-functions}.
Notably, intercommunicator collectives are {\em not} defined for
\mpifunc{MPI\_\-SCAN} (because the MPI standard does not define it),
\mpifunc{MPI\_\-ALLGATHERV}, and
\mpifunc{MPI\_\-EXSCAN}.\footnote{These two functions were
  unfortunately overlooked and forgotten about when Open MPI v7.1 was
  frozen for release.}

\begin{table}[htbp]
  \centering
  \begin{tabular}{|lll|}
    \hline
    \multicolumn{3}{|c|}{Supported Functions} \\
    \hline
    \hline
    \mpifunc{MPI\_\-ALLGATHER} &
    \mpifunc{MPI\_\-ALLGATHERV} &
    \mpifunc{MPI\_\-ALLTOALL} \\
%
    \mpifunc{MPI\_\-ALLTOALLV} &
    \mpifunc{MPI\_\-ALLTOALLW} &
    \mpifunc{MPI\_\-ALLREDUCE} \\
%
    \mpifunc{MPI\_\-REDUCE\_\-SCATTER} &
    \mpifunc{MPI\_\-GATHER} &
    \mpifunc{MPI\_\-GATHERV} \\
%
    \mpifunc{MPI\_\-REDUCE} &
    \mpifunc{MPI\_\-BCAST} &
    \mpifunc{MPI\_\-SCATTER} \\
%
    \mpifunc{MPI\_\-SCATTERV} &
    \mpifunc{MPI\_\-BARRIER} &
    ~ \\
    \hline
  \end{tabular}
  \caption{Supported MPI-2 intercommunicator collective functions.}
  \label{tbl:mpi-details-intercomm-collective-functions}
\end{table}

\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{External Interfaces}

The external interfaces chapter lists several different major topics.
Open MPI's support for these topics is summarized in
Table~\ref{tbl:mpi-details-ext-int-summary}, and the exact list of
functions that are supported is listed
in~\ref{tbl:mpi-details-ext-int-functions}.

\begin{table}[htbp]
  \centering
  \begin{tabular}{|c|p{3in}|}
    \hline
    \multicolumn{1}{|c|}{Supported} &
    \multicolumn{1}{|c|}{Description} \\
    \hline
    \hline
%
    \supportno & Generalized requests \\
%
    \supportno & Associating information with \mpitype{MPI\_\-Status} \\
%
    \supportyes & Naming objects \\
%
    \supportno & Error classes \\
    \supportno & Error codes \\
    \supportyes & Error handlers \\
%
    \supportyes & Decoding a datatype \\
%
    \supportyes & MPI and threads \\
%
    \supportyes & New attribute caching functions \\
%
    \supportyes & Duplicating a datatype \\
    \hline
  \end{tabular}
  \caption{Major topics in the MPI-2 chapter ``External Interfaces'',
    and Open MPI's level of support.}
  \label{tbl:mpi-details-ext-int-summary}
\end{table}

\begin{table}[htbp]
  \centering
  \begin{tabular*}{\textwidth}{llllll|}
    \hline
    \multicolumn{6}{|c|}{Supported Functions} \\
    \hline \hline
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-COMM\_\-SET\_\-NAME}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-TYPE\_\-SET\_\-NAME}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-WIN\_\-SET\_\-NAME}} \\
%
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-COMM\_\-GET\_\-NAME}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-TYPE\_\-GET\_\-NAME}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-WIN\_\-GET\_\-NAME}} \\
%
    \hline
%
    \hline
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-COMM\_\-CREATE\_\-ERRHANDLER}} &
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-WIN\_\-CREATE\_\-ERRHANDLER}} \\
%
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-COMM\_\-GET\_\-ERRHANDLER}} &
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-WIN\_\-GET\_\-ERRHANDLER}} \\
%
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-COMM\_\-SET\_\-ERRHANDLER}} &
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-WIN\_\-SET\_\-ERRHANDLER}} \\
%
    \hline
%
    \hline
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-TYPE\_\-GET\_\-CONTENTS}} &
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-INIT\_\-THREAD}} \\

%
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-TYPE\_\-GET\_\-ENVELOPE}} &
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-QUERY\_\-THREAD}} \\
%
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-TYPE\_\-GET\_\-EXTENT}} &
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-IS\_\-THREAD\_\-MAIN}} \\
%
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-TYPE\_\-GET\_\-TRUE\_\-EXTENT}} &
    \multicolumn{3}{|l|}{\mpifunc{MPI\_\-TYPE\_\-DUP}} \\
%
    \hline
%
    \hline
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-COMM\_\-CREATE\_\-KEYVAL}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-TYPE\_\-CREATE\_\-KEYVAL}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-WIN\_\-CREATE\_\-KEYVAL}} \\
%
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-COMM\_\-FREE\_\-KEYVAL}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-TYPE\_\-FREE\_\-KEYVAL}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-WIN\_\-FREE\_\-KEYVAL}} \\
%
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-COMM\_\-DELETE\_\-ATTR}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-TYPE\_\-DELETE\_\-ATTR}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-WIN\_\-DELETE\_\-ATTR}} \\
%
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-COMM\_\-GET\_\-ATTR}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-TYPE\_\-GET\_\-ATTR}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-WIN\_\-GET\_\-ATTR}} \\
%
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-COMM\_\-SET\_\-ATTR}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-TYPE\_\-SET\_\-ATTR}} &
    \multicolumn{2}{|l|}{\mpifunc{MPI\_\-WIN\_\-SET\_\-ATTR}} \\
%
    \hline
  \end{tabular*}
  \caption{Supported MPI-2 external interface functions, grouped by
    function.}
  \label{tbl:mpi-details-ext-int-functions}
\end{table}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{I/O}

MPI-IO support is provided by including the ROMIO package from Argonne
National Labs,\footnote{\url{http://www.mcs.anl.gov/romio/}} version
1.2.5.1.
%
The Open MPI wrapper compilers (\icmd{mpicc}, \icmd{mpiCC}/\icmd{mpic++},
and \icmd{mpif77}) will automatically provide all the necessary flags
to compile and link programs that use ROMIO function calls.

Although the ROMIO group at Argonne has included support for Open MPI in
their package, there are still a small number of things that the Open MPI
Team had to do to make ROMIO compile and install properly with
Open MPI.  As such, if you try to install the ROMIO package manually
with Open MPI, you will experience some difficulties.

There are some important limitations to ROMIO that are discussed in
the \file{romio/README} file.
%
One limitation that is not currently listed in the ROMIO README file
is that atomic file access will not work with AFS.  This is because of
file locking problems with AFS (i.e., AFS iteself does not support
file locking).  The ROMIO test program \cmd{atomicity} will fail if
you specify an output file on AFS.

Additionally, ROMIO does not support the following Open MPI functionality:

\begin{itemize}
\item Open MPI MPI-2 datatypes cannot be used with ROMIO; ROMIO makes the
  fundamental assumption that MPI-2 datatypes are built upon MPI-1
  datatypes.  Open MPI builds MPI-2 datatypes natively -- ROMIO cannot
  presently handle this case.
  
  This will hopefully be fixed in some future release of ROMIO.  The
  ROMIO test programs \cmd{coll\_\-test}, \cmd{fcoll\_\-test},
  \cmd{large\_\-array}, and \cmd{coll\_\-perf} will fail because they
  use the MPI-2 datatype \mpidatatype{MPI\_\-DARRAY}.
\end{itemize}

Please see the sections ``ROMIO Users Mailing List'' and ``Reporting
Bugs'' in \file{romio/README} for how to submit questions and bug
reports about ROMIO (that do not specifically pertain to Open MPI).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Language Bindings}

Open MPI provides C, C++, and Fortran 77 bindings for all supported MPI-2
functions, types, and constants.  Open MPI does not provide a Fortran 90
module.
%
However, it is possible to use the Fortran 77 bindings with a Fortran
90 compiler by specifying the F90 compiler as your Fortran compiler
when configuring/compiling Open MPI.  See the Open MPI Installation
Guide~\cite{lamteam03:_lam_mpi_install_guide} for more details.

The C++ bindings include support for the C++ only \mpitype{MPI::BOOL},
\mpitype{MPI::COMPLEX}, \mpitype{MPI::DOUBLE\_\-COMPLEX}, and
\mpitype{MPI::LONG\_\-DOUBLE\_\-COMPLEX} datatypes.

Note that there are some issues with using MPI and Fortran 90
together.  See the F90 / C++ chapter in the MPI-2
standard~\cite{geist96:_mpi2_lyon} for more information on using MPI
with Fortran 90.

As mentioned in Section~\ref{sec:mpi-details-mpi1-language}, profiling
support is available in all three languages (if Open MPI was compiled
with profiling support).  The \icmd{ompi\_info} command can be used to
see if profiling support was included in Open MPI.