openmpi/doc/user/commands.tex

% -*- latex -*-
%
% Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
%                         University Research and Technology
%                         Corporation.  All rights reserved.
% Copyright (c) 2004-2005 The University of Tennessee and The University
%                         of Tennessee Research Foundation.  All rights
%                         reserved.
% Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, 
%                         University of Stuttgart.  All rights reserved.
% Copyright (c) 2004-2005 The Regents of the University of California.
%                         All rights reserved.
% $COPYRIGHT$
% 
% Additional copyrights may follow
% 
% $HEADER$
%

\chapter{Open MPI Command Quick Reference}
\label{sec:commands}

This section is intended to provide a quick reference of the major
Open MPI commands.  Each command also has its own manual page which
typically provides more detail than this document.

{\Huge JMS needs total overhaul}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamboot} Command}
\label{sec:commands-lamboot}

The \cmd{lamboot} command is used to start the Open MPI run-time
environment (RTE).  \cmd{lamboot} is typically the first command used
before any other Open MPI command (notable exceptions are the wrapper
compilers, which do not require the Open MPI RTE, and \cmd{mpiexec} which
can launch its own Open MPI universe).  \cmd{lamboot} can use any of the
available \kind{boot} SSI modules; Section~\ref{sec:lam-ssi-boot}
details the requirements and operations of each of the \kind{boot} SSI
modules that are included in the Open MPI distribution.

Common arguments that are used with the \cmd{lamboot} command are:

\begin{itemize}
\item \cmdarg{-b}: When used with the \boot{rsh} boot module, the
  ``fast'' boot algorithm is used which can noticeably speed up the
  execution time of \cmd{lamboot}.  It can also be used where remote
  shell agents cannot provide output from remote nodes (e.g., in a
  Condor environment).  Specifically, the ``fast'' algorithm assumes
  that the user's shell on the remote node is the same as the shell on
  the node where \cmd{lamboot} was invoked.
  
\item \cmdarg{-d}: Print debugging output.  This will print a {\em
    lot} of output, and is typically only necessary if \cmd{lamboot}
  fails for an unknown reason.  The output is forwarded to standard
  out as well as either \file{/tmp} or syslog facilities.  The amount of
  data produced can fill these filesystems, leading to general system
  problems.

\item \cmdarg{-l}: Use local hostname resolution instead of
  centralized lookups.  This is useful in environments where the same
  hostname may resolve to different IP addresses on different nodes
  (e.g., clusters based on Finite Neighborhood Networks\footnote{See
    \url{http://www.aggregate.org/} for more details.}).
  
\changebegin{7.1}

\item \cmdarg{-prefix $<$lam/install/path$>$}: Use the Open MPI
  installation specified in the $<$lam/install/path$>$ - where
  $<$lam/install/path$>$ is the top level directory where Open MPI is
  installed. This is typically used when a user has multiple Open MPI
  installations and want to switch between them without changing the
  dot files or PATH environment variable.
  
  This option is not compatible with Open MPI versions prior to 7.1.

\changeend{7.1}

\item \cmdarg{-s}: Close the \file{stdout} and \file{stderr} of the
  locally-launched Open MPI daemon (they are normally left open).  This is
  necessary when invoking \cmd{lamboot} via a remote agent such as
  \cmd{rsh} or \cmd{ssh}.
  
\item \cmdarg{-v}: Print verbose output.  This is useful to show
  progress during \cmd{lamboot}'s progress.  Unlike \cmdarg{-d},
  \cmdarg{-v} does not forward output to a file or syslog.

\item \cmdarg{-x}: Run the Open MPI RTE in fault-tolerant mode.
  
\item \cmdarg{$<$filename$>$}: The name of the boot schema file.  Boot
  schemas, while they can be as simple as a list of hostnames, can
  contain additional information and are discussed in detail
  in Sections~\ref{sec:getting-started-hostfile} and
  ~\ref{sec:lam-ssi-boot-schema},
  pages~\pageref{sec:getting-started-hostfile}
  and~\pageref{sec:lam-ssi-boot-schema}, respectively.
\end{itemize}

Booting the Open MPI RTE is where most users (particularly first-time
users) encounter problems.  Each \kind{boot} module has its own
specific requirements and prerequisites for success.  Although
\cmd{lamboot} typically prints detailed messages when errors occur,
users are strongly encouraged to read Section~\ref{sec:lam-ssi-boot}
for the details of the \kind{boot} module that they will be using.
Additionally, the \cmdarg{-d} switch should be used to examine exactly
what is happening to determine the actual source of the problem --
many problems with \cmd{lamboot} come from the operating system or the
user's shell setup; not from within Open MPI itself.

The most common \cmd{lamboot} example simply uses a hostfile to launch
across an \cmd{rsh}/\cmd{ssh}-based cluster of nodes (the
``\cmdarg{-ssi boot rsh}'' is not technically necessary here, but it
is specified to make this example correct in all environments):

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamboot -v -ssi boot rsh hostfile

Open MPI 7.0/MPI 2 C++/ROMIO - Indiana University

n0<1234> ssi:boot:base:linear: booting n0 (node1.cluster.example.com)
n0<1234> ssi:boot:base:linear: booting n1 (node2.cluster.example.com)
n0<1234> ssi:boot:base:linear: booting n2 (node3.cluster.example.com)
n0<1234> ssi:boot:base:linear: booting n3 (node4.cluster.example.com)
n0<1234> ssi:boot:base:linear: finished
\end{lstlisting}
% Stupid emacs mode: $

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Multiple Sessions on the Same Node}

In some cases (such as in batch-regulated environments), it is
desirable to allow multiple universes owned by the same on the same
node.  The \ienvvar{TMPDIR},
\ienvvar{Open MPI\_\-MPI\_\-SESSION\_\-PREFIX}, and
\ienvvar{Open MPI\_\-MPI\_\-SESSION\_\-SUFFIX} environment variables can be
used to effect this behavior.  The main issue is the location of Open MPI's
session directory; each node in a Open MPI universe has a session directory
in a well-known location in the filesystem that identifies how to
contact the Open MPI daemon on that node.  Multiple Open MPI universes can
simultaneously co-exist on the same node as long as they have
different session directories.

Open MPI recognizes several batch environments and automatically adapts the
session directory to be specific to a batch job.  Hence, if the batch
scheduler allocates multiple jobs from the same user to the same node,
Open MPI will automatically do the ``right thing'' and ensure that the Open MPI
universes from each job will not collide.
%
Sections~\ref{sec:misc-batch} and~\ref{sec:misc-session-directory}
(starting on page~\pageref{sec:misc-batch}) discuss these issues in
detail.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Avoiding Running on Specific Nodes}
\label{sec:commands-lamboot-no-schedule}
\index{no-schedule boot schema attribute@\cmd{no-schedule} boot
  schema attribute}

Once the Open MPI universe is booted, processes can be launched on any
node.  The \cmd{mpirun}, \cmd{mpiexec}, and \cmd{lamexec} commands are
most commonly used to launch jobs in the universe, and are typically
used with the \cmdarg{N} and \cmdarg{C} nomenclatures (see the description of
\cmd{mpirun} in Section~\ref{sec:commands-mpirun} for details on the
\cmdarg{N} and \cmdarg{C} nomenclature) which launch jobs on all schedulable
nodes and CPUs in the Open MPI universe, respectively.  While finer-grained
controls are available through \cmd{mpirun} (etc.), it can be
convenient to simply mark some nodes as ``non-schedulable,'' and
therefore avoid having \cmd{mpirun} (etc.) launch executables on those
nodes when using \cmdarg{N} and \cmdarg{C} nomenclature.

For example, it may be convenient to boot a Open MPI universe that includes
a controller node (e.g., a desktop workstation) and a set of worker
nodes.  In this case, it is desirable to mark the desktop workstation
as ``non-scheduable'' so that Open MPI will not launch executables there
(by default).  Consider the following boot schema:

\lstset{style=lam-shell}
\begin{lstlisting}
# Mark my_workstation as ``non-schedulable''
my_workstation.office.example.com schedule=no
# All the other nodes are, by default, schedulable
node1.cluster.example.com
node2.cluster.example.com
node3.cluster.example.com
node4.cluster.example.com
\end{lstlisting}

Booting with this schema allows the convenienve of:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpirun C my_mpi_program
\end{lstlisting}
% stupid emacs mode: $

\noindent which will only run \cmd{my\_\-mpi\_\-program} on the four
cluster nodes (i.e., not the workstation).  
%
Note that this behavior {\em only} applies to the \cmdarg{C} and \cmdarg{N}
designations; Open MPI will always allow execution on any node when using
the \cmdarg{nX} or \cmdarg{cX} notation:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpirun c0 C my_mpi_program
\end{lstlisting}
% stupid emacs mode: $

\noindent which will run \cmd{my\_\-mpi\_\-program} on all five nodes
in the Open MPI universe.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamcheckpoint} Command}
\label{sec:commands-lamcheckpoint}

\changebegin{7.1}

The \cmd{lamcheckpoint} command is provided to checkpoint a MPI
application.  One of the arguments to \cmd{lamcheckpoint} is the name
of the checkpoint/restart module (which can be either one of
\crssi{blcr} and \crssi{self}).  Additional arguments to
\cmd{lamcheckpoint} depend of the selected checkpoint/restart module.
The name of the module can be specified by passing the \crssi{cr} SSI
parameter.

Common arguments that are used with the \cmd{lamcheckpoint} command
are:

\begin{itemize}
\item \cmdarg{-ssi}: Just like with \cmd{mpirun}, the \cmdarg{-ssi}
  flag can be used to pass key=value pairs to Open MPI.  Indeed, it is
  required to pass at least one SSI parameter: \ssiparam{cr},
  indicating which \kind{cr} module to use for checkpointing.
  
\item \cmdarg{-pid}: Indicate the PID of \cmd{mpirun} to checkpoint.
\end{itemize}

\noindent Notes:

\begin{itemize}
\item If the \crssi{blcr} \kind{cr} module is selected, the name of
  the directory for storing the checkpoint files and the PID of
  \cmd{mpirun} should be passed as SSI parameters to
  \cmd{lamcheckpoint}.
  
\item If the \crssi{self} \kind{cr} module is selected, the PID of
  \cmd{mpirun} should be passed via the \cmdarg{-pid} parameter.
\end{itemize}

\changeend{7.1}

See Section~\ref{sec:mpi-ssi-cr} for more detail about the
checkpoint/restart capabilities of Open MPI, including details about
the \crssi{blcr} and \crssi{self} \kind{cr} modules.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamclean} Command}
\label{sec:commands-lamclean}

The \cmd{lamclean} command is provided to clean up the Open MPI universe.
It is typically only necessary when MPI processes terminate ``badly,''
and potentially leave resources allocated in the Open MPI universe (such as
MPI-2 published names, processes, or shared memory).  The
\cmd{lamclean} command will kill {\em all} processes running in the
Open MPI universe, and free {\em all} resources that were associated with
them (including unpublishing MPI-2 dynamicly published names).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamexec} Command}
\label{sec:commands-lamexec}

The \cmd{lamexec} command is similar to \cmd{mpirun} but is used for
non-MPI programs.  For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamexec N uptime
  5:37pm  up 21 days, 23:49,  5 users,  load average: 0.31, 0.26, 0.25
  5:37pm  up 21 days, 23:49,  2 users,  load average: 0.01, 0.00, 0.00
  5:37pm  up 21 days, 23:50,  3 users,  load average: 0.01, 0.00, 0.00
  5:37pm  up 21 days, 23:50,  2 users,  load average: 0.87, 0.81, 0.80
\end{lstlisting}
% Stupid emacs: $

Most of the parameters and options that are available to \cmd{mpirun}
are also available to \cmd{lamexec}.  See the \cmd{mpirun} description
in Section~\ref{sec:commands-mpirun} for more details.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamgrow} Command}
\label{sec:commands-lamgrow}

The \cmd{lamgrow} command adds a single node to the Open MPI universe.  It
must use the same \kind{boot} module that was used to initially boot
the Open MPI universe.  \cmd{lamgrow} must be run from a node already in
the Open MPI universe.  Common parameters include:

\begin{itemize}
\item \cmdarg{-v}: Verbose mode.
  
\item \cmdarg{-d}: Debug mode; enables a {\em lot} of diagnostic
  output.

\item \cmdarg{-n $<$nodeid$>$}: Assign the new host the node ID
  \cmdarg{nodeid}.   \cmdarg{nodeid} must be an unused node ID.  If
  \cmdarg{-n} is not specified, Open MPI will find the lowest node ID that
  is not being used.
  
\item \cmdarg{-no-schedule}: Has the same effect as putting ``{\tt
    no\_\-schedule=yes}'' in the boot schema.  This means that the
  \cmdarg{C} and \cmdarg{N} expansion used in \cmd{mpirun} and \cmd{lamexec}
  will not include this node.

\item \cmdarg{-ssi $<$key$>$ $<$value$>$}: Pass in SSI parameter
  \cmdarg{key} with the value \cmdarg{value}.

\item \cmdarg{$<$hostname$>$}: The name of the host to expand the
  universe to.
\end{itemize}

For example, the following adds the node \host{blinky} to the existing
Open MPI universe using the \boot{rsh} boot module:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamgrow -ssi boot rsh blinky.cluster.example.com
\end{lstlisting}
% Stupid emacs: $

Note that \cmd{lamgrow} cannot grow a Open MPI universe that only contains
one node that has an IP address of 127.0.0.1 (e.g., if \cmd{lamboot}
was run with the default boot schema that only contains the name
\host{localhost}).  In this case, \cmd{lamgrow} will print an error
and abort without adding the new node.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamhalt} Command}
\label{sec:commands-lamhalt}

The \cmd{lamhalt} command is used to shut down the Open MPI RTE.
Typically, \cmd{lamhalt} can simply be run with no command line
parameters and it will shut down the Open MPI RTE.  Optionally, the
\cmdarg{-v} or \cmdarg{-d} arguments can be used to make \cmd{lamhalt}
be verbose or extremely verbose, respectively.

There are a small number of cases where \cmd{lamhalt} will fail.  For
example, if a Open MPI daemon becomes unresponsive (e.g., the daemon was
killed), \cmd{lamhalt} may fail to shut down the entire Open MPI universe.
It will eventually timeout and therefore complete in finite time, but
you may want to use the last-resort \cmd{lamwipe} command (see
Section~\ref{sec:commands-lamwipe}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{laminfo} Command}
\label{sec:commands-laminfo}

The \cmd{laminfo} command can be used to query the capabilities of the
Open MPI installation.  Running \cmd{laminfo} with no parameters shows
a prettyprint summary of information.  Using the \cmdarg{-parsable}
command line switch shows the same summary information, but in a
format that should be relatively easy to parse with common unix tools
such as \cmd{grep}, \cmd{cut}, \cmd{awk}, etc.

\cmd{laminfo} supports a variety of command line options to query for
specific information.  The \cmdarg{-h} option shows a complete listing
of all options.  Some of the most common options include:

\begin{itemize}
\item \cmdarg{-arch}: Show the architecture that Open MPI was configured
  for.

\item \cmdarg{-path}: Paired with a second argument, display various
  paths relevant to the Open MPI installation.  Valid second arguments
  include: 

  \begin{itemize}
  \item \cmdarg{prefix}: Main installation prefix
  \item \cmdarg{bindir}: Where the Open MPI executables are located
  \item \cmdarg{libdir}: Where the Open MPI libraries are located
  \item \cmdarg{incdir}: Where the Open MPI include files are located
  \item \cmdarg{pkglibdir}: Where dynamic SSI modules are
    installed\footnote{Dynamic SSI modules are not supported in
      Open MPI 7.0, but will be supported in future versions.}
  \item \cmdarg{sysconfdir}: Where the Open MPI help files are located
  \end{itemize}

\item \cmdarg{-version}: Paired with two addition options, display the
  version of either Open MPI or one or more SSI modules.  The first
  argument identifies what to report the version of, and can be any of
  the following:

  \begin{itemize}
  \item \cmdarg{lam}: Version of Open MPI
  \item \cmdarg{boot}: Version of all boot modules
  \item \cmdarg{boot:module}: Version of a specific boot module
  \item \cmdarg{coll}: Version of all coll modules
  \item \cmdarg{coll:module}: Version of a specific coll module
  \item \cmdarg{cr}: Version of all cr modules
  \item \cmdarg{cr:module}: Version of a specific cr module
  \item \cmdarg{rpi}: Version of all rpi modules
  \item \cmdarg{rpi:module}: Version of a specific rpi module
  \end{itemize}

  The second argument specifies the scope of the version number to
  display -- whether to show the entire version number string, or just
  one component of it:

  \begin{itemize}
  \item \cmdarg{full}: Display the entire version number string
  \item \cmdarg{major}: Display the major version number
  \item \cmdarg{minor}: Display the minor version number
  \item \cmdarg{release}: Display the release version number
  \item \cmdarg{alpha}: Display the alpha version number
  \item \cmdarg{beta}: Display the beta version number
  \item \cmdarg{svn}: Display the SVN version number\footnote{The
      value will either be 0 (not built from SVN), 1 (built from a
      Subverstion checkout) or a date encoded in the form YYYYMMDD
      (built from a nightly tarball on the given date)}

  \end{itemize}

\changebegin{7.1}

\item \cmdarg{-param}: Paired with two additional arguments, display
  the SSI parameters for a given type and/or module.  The first
  argument can be any of the valid SSI types or the special name
  ``base,'' indicating the SSI framework itself.  The second argument
  can be any valid module name.
  
  Additionally, either argument can be the wildcard ``any'' which
  will match any valid SSI type and/or module.

\changeend{7.1}
\end{itemize}

Multiple options can be combined to query several attributes at once:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ laminfo -parsable -arch -version lam major -version rpi:tcp full -param rpi tcp
version:lam:7
ssi:boot:rsh:version:ssi:1.0
ssi:boot:rsh:version:api:1.0
ssi:boot:rsh:version:module:7.0
arch:i686-pc-linux-gnu
ssi:rpi:tcp:param:rpi_tcp_short:65536
ssi:rpi:tcp:param:rpi_tcp_sockbuf:-1
ssi:rpi:tcp:param:rpi_tcp_priority:20
\end{lstlisting}
% Stupid emacs: $

Note that three version numbers are returned for the \rpi{tcp} module.
The first (\cmdarg{ssi}) indicates the overall SSI version that the
module conforms to, the second (\cmdarg{api}) indicates what version
of the \kind{rpi} API the module conforms to, and the last
(\cmdarg{module}) indicates the version of the module itself.

Running \cmd{laminfo} with no arguments provides a wealth of
information about your Open MPI installation (we ask for this output
when reporting problems to the Open MPI general user's mailing list --
see Section \ref{troubleshooting:mailing-lists} on page
\pageref{troubleshooting:mailing-lists}).  Most of the output fields
are self-explanitory; two that are worth explaining are:

\begin{itemize}
\item Debug support: This indicates whether your Open MPI installation was
  configured with the \confflag{with-debug} option.  It is generally
  only used by the Open MPI Team for development and maintenance of Open MPI
  itself; it does {\em not} indicate whether user's MPI applications
  can be debugged (specifically: user's MPI applications can {\em
    always} be debugged, regardless of this setting).  This option
  defaults to ``no''; users are discouraged from using this option.
  See the Install Guide for more information about
  \confflag{with-debug}.
  
\item Purify clean: This indicates whether your Open MPI installation was
  configured with the \confflag{with-purify} option.  This option is
  necessary to prevent a number of false positives when using
  memory-checking debuggers such as Purify, Valgrind, and bcheck.  It
  is off by default because it can cause slight performance
  degredation in MPI applications.  See the Install Guide for more
  information about \confflag{with-purify}.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamnodes} Command}
\label{sec:commands-lamnodes}

Open MPI was specifically designed to abstract away hostnames once
\cmd{lamboot} has completed successfully.  However, for various
reasons (usually related to system-administration concerns, and/or for
creating human-readable reports), it can be desirable to retrieve the
hostnames of Open MPI nodes long after \icmd{lamboot}.

The command \cmd{lamnodes} can be used for this purpose.  It accepts
both the \cmdarg{N} and \cmdarg{C} syntax from \cmd{mpirun}, and will return the
corresponding names of the specified nodes.  For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamnodes N
\end{lstlisting}
% Stupid emacs: $

\noindent will return the node that each CPU is located on, the
hostname of that node, the total number of CPUs on each, and any flags
that are set on that node.  Specific nodes can also be queried:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamnodes n0,3
\end{lstlisting}
% Stupid emacs: $

\noindent will return the node, hostname, number of CPUs, and flags on
n0 and n3.

Command line arguments can be used to customize the output of
\cmd{lamnodes}.  These include:

\begin{itemize}
\item \cmdarg{-c}: Suppress printing CPU counts
\item \cmdarg{-i}: Print IP addresses instead of IP names
\item \cmdarg{-n}: Suppress printing Open MPI node IDs
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamrestart} Command}

The \cmd{lamrestart} can be used to restart a previously-checkpointed
MPI application.  The arguments to \cmd{lamrestart} depend on the
selected checkpoint/restart module.  Regardless of the
checkpoint/restart module used, invoking \cmd{lamrestart} results in a
new \cmd{mpirun} being launched.

The SSI parameter \ssiparam{cr} must be used to specify which
checkpoint/restart module should be used to restart the application.
Currently, only two values are possible: \ssiparam{blcr} and
\ssiparam{self}.

\begin{itemize}
\item If the \crssi{blcr} module is selected, the SSI parameter
  \issiparam{cr\_\-blcr\_\-context\_\-file} should be used to pass in
  the filename of the context file that was created during a pevious
  successful checkpoint.  For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamrestart -ssi cr blcr -ssi cr_blcr_context_file filename
\end{lstlisting}
% Stupid emacs: $
  
\item If the \crssi{self} module is selected, the SSI parameter
  \issiparam{cr\_\-restart\_\-args} must be passed with the arguments
  to be passed to \cmd{mpirun} to restart the application.  For
  example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamrestart -ssi cr self -ssi cr_restart_args "args to mpirun"
\end{lstlisting}
% Stupid emacs: $
\end{itemize}

See Section~\ref{sec:mpi-ssi-cr} for more detail about the
checkpoint/restart capabilities of Open MPI, including details about
the \crssi{blcr} and \crssi{self} \kind{cr} modules.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The \icmd{lamshrink} Command}
\label{sec:commands-lamshrink}

The \cmd{lamshrink} command is used to remove a node from a Open MPI
universe:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamshrink n3
\end{lstlisting}
% Stupid emacs: $

\noindent removes node n3 from the Open MPI universe.  Note that all nodes
with ID's greater than 3 will not have their ID's reduced by one -- n3
simply becomes an empty slot in the Open MPI universe.  \cmd{mpirun} and
\cmd{lamexec} will still function correctly, even when used with \cmdarg{C}
and \cmdarg{N} notation -- they will simply skip the n3 since there is no
longer an operational node in that slot.

Note that the \cmd{lamgrow} command can optionally be used to fill the
empty slot with a new node.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{mpicc}, \icmd{mpiCC} / \icmd{mpic++}, and
  \icmd{mpif77} Commands}
\label{sec:commands-wrappers}
\index{wrapper compilers}

Compiling MPI applications can be a complicated process because the
list of compiler and linker flags required to successfully compile and
link a Open MPI application not only can be quite long, it can change
depending on the particular configuration that Open MPI was installed with.
For example, if Open MPI includes native support for Myrinet hardware, the
\cmdarg{-lgm} flag needs to be used when linking MPI executables.

To hide all this complexity, ``wrapper'' compilers are provided that
handle all of this automatically.  They are called ``wrapper''
compilers because all they do is add relevant compiler and linker
flags to the command line before invoking the real back-end compiler
to actually perform the compile/link.  Most command line arugments are
passed straight through to the back-end compiler without modification.

Therefore, to compile an MPI application, use the wrapper compilers
exactly as you would use the real compiler.  For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpicc -O -c main.c
shell$ mpicc -O -c foo.c
shell$ mpicc -O -c bar.c
shell$ mpicc -O -o main main.o foo.o bar.o
\end{lstlisting}

This compiles three C source code files and links them together into a
single executable.  No additional \cmdarg{-I}, \cmdarg{-L}, or
\cmdarg{-l} arguments are required.

The main exceptions to what flags are not passed through to the
back-end compiler are:

\begin{itemize}
\item \cmdarg{-showme}: Used to show what the wrapper compiler would
  have executed.  This is useful to see the full compile/link line
  would have been executed.  For example (your output may differ from
  what is shown below, depending on your installed Open MPI
  configuration):

  \lstset{style=lam-cmdline}
  \begin{lstlisting}
shell$ mpicc -O -c main.c -showme
gcc -I/usr/local/lam/include -pthread -O -c foo.c
  \end{lstlisting}
% Stupid emacs mode: $
  \lstset{style=lam-cmdline}
  \begin{lstlisting}
# The output line shown below is word wrapped in order to fit nicely in the document margins
shell$ mpicc -O -o main main.o foo.o bar.o -showme
gcc -I/usr/local/lam/include -pthread -O -o main main.o foo.o bar.o \
-L/usr/local/lam/lib -llammpio -lpmpi -llamf77mpi -lmpi -llam -lutil \
-pthread 
  \end{lstlisting}
% Stupid emacs mode: $

  \changebegin{7.1}

  Two notable sub-flags are:

  \begin{itemize}
  \item \cmdarg{-showme:compile}: Show only the compile flags,
    suitable for substitution into \envvar{CFLAGS}.
    
  \lstset{style=lam-cmdline}
  \begin{lstlisting}
shell$ mpicc -O -c main.c -showme:compile
-I/usr/local/lam/include -pthread
  \end{lstlisting}
% Stupid emacs mode: $

  \item \cmdarg{-showme:link}: Show only the linker flags (which are
    actually \envvar{LDFLAGS} and \envvar{LIBS} mixed together),
    suitable for substitution into \envvar{LIBS}.

  \lstset{style=lam-cmdline}
  \begin{lstlisting}
shell$ mpicc -O -o main main.o foo.o bar.o -showme:link
-L/usr/local/lam/lib -llammpio -lpmpi -llamf77mpi -lmpi -llam -lutil -pthread 
  \end{lstlisting}
% Stupid emacs mode: $

  \end{itemize}

  \changeend{7.1}

\item \cmdarg{-lpmpi}: When compiling a user MPI application, the
  \cmdarg{-lpmpi} argument is used to indicate that MPI profiling
  support should be included.  The wrapper compiler may alter the
  exact placement of this argument to ensure that proper linker
  dependency semantics are preserved.
\end{itemize}

\changebegin{7.1}
Neither the compiler nor linker flags can be overridden at run-time.
The back-end compiler, however, can be.  Environment variables can be
used for this purpose:

\begin{itemize}
\item \ienvvar{Open MPIMPICC} (deprecated name: \idepenvvar{Open MPIHCC}):
  Overrides the default C compiler in the \cmd{mpicc} wrapper
  compiler.
  
\item \ienvvar{Open MPIMPICXX} (deprecated name: \idepenvvar{Open MPIHCP}):
  Overrides the default C compiler in the \cmd{mpicc} wrapper
  compiler.

\item \ienvvar{Open MPIMPIF77} (deprecated name: \idepenvvar{Open MPIHF77}):
  Overrides the default C compiler in the \cmd{mpicc} wrapper
  compiler.
\end{itemize}

For example (for Bourne-like shells):

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ Open MPIPICC=cc
shell$ export Open MPIMPICC
shell$ mpicc my_application.c -o my_application
\end{lstlisting}
% Stupid emacs mode: $

For csh-like shells:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell% setenv Open MPIPICC cc
shell% mpicc my_application.c -o my_application
\end{lstlisting}

All this being said, it is {\em strongly} recommended to use the
wrapper compilers -- and their default underlying compilers -- for all
compiling and linking of MPI applications.  Strange behavior can occur
in MPI applications if Open MPI was configured and compiled with one
compiler and then user applications were compiled with a different
underlying compiler, to include: failure to compile, failure to link,
seg faults and other random bad behavior at run-time.

Finally, note that the wrapper compilers only add all the
Open MPI-specific flags when a command-line argument that does not
begin with a dash (``-'') is present.  For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpicc
gcc: no input files
shell$ mpicc --version
gcc (GCC) 3.2.2 (Mandrake Linux 9.1 3.2.2-3mdk)
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
\end{lstlisting}

\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Deprecated Names}

Previous versions of Open MPI used the names \idepcmd{hcc},
\idepcmd{hcp}, and \idepcmd{hf77} for the wrapper compilers.  While
these command names still work (they are simply symbolic links to the
real wrapper compilers \cmd{mpicc}, \cmd{mpiCC}/\cmd{mpic++}, and
\cmd{mpif77}, respectively), their use is deprecated.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{mpiexec} Command}
\label{sec:commands-mpiexec}

The \cmd{mpiexec} command is used to launch MPI programs.  It is
similar to, but slightly different than, \cmd{mpirun}.\footnote{The
  reason that there are two methods to launch MPI executables is
  because the MPI-2 standard suggests the use of \cmd{mpiexec} and
  provides standardized command line arguments.  Hence, even though
  Open MPI already had an \cmd{mpirun} command to launch MPI executables,
  \cmd{mpiexec} was added to comply with the standard.}  Although
\cmd{mpiexec} is simply a wrapper around other Open MPI commands (including
\cmd{lamboot}, \cmd{mpirun}, and \cmd{lamhalt}), it ties their
functionality together and provides a unified interface for launching
MPI processes. 
%
Specifically, \cmd{mpiexec} offers two features from command line
flags that require multiple steps when using other Open MPI commands:
launching MPMD MPI processes and launching MPI processes when there is
no existing Open MPI universe.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{General Syntax}

The general form of \cmd{mpiexec} commands is:

\lstset{style=lam-cmdline}
\begin{lstlisting}
  mpiexec [global_args] local_args1 [: local_args2 [...]]
\end{lstlisting}

Global arguments are applied to all MPI processes that are launched.
They must be specified before any local arguments.  Common global
arguments include:

\begin{itemize}
\item \cmdarg{-boot}: Boot the Open MPI RTE before launching the MPI
  processes.

\item \cmdarg{-boot-args $<$args$>$}: Pass \cmdarg{$<$args$>$} to the
  back-end \cmd{lamboot}.  Implies \cmdarg{-boot}.

\item \cmdarg{-machinefile $<$filename$>$}: Specify {\tt
    $<$filename$>$} as the boot schema to use when invoking the
    back-end \cmd{lamboot}.  Implies \cmdarg{-boot}.

\changebegin{7.1}
\item \cmdarg{-prefix $<$lam/install/path$>$}: Use the Open MPI
  installation specified in the $<$lam/install/path$>$ - where
  $<$lam/install/path$>$ is the top level directory where Open MPI is
  ``installed''. This is typically used when a user has multiple
  Open MPI installations and want to switch between them without
  changing the dot files or PATH environment variable.  This option is
  not compatible with Open MPI versions prior to 7.1.
\changeend{7.1}

\item \cmdarg{-ssi $<$key$>$ $<$value$>$}: Pass the SSI {\tt
    $<$key$>$} and {\tt $<$value$>$} arguments to the back-end
    \cmd{mpirun} command.
\end{itemize}

Local arguments are specific to an individual MPI process that will be
launched.  They are specified along with the executable that will be
launched.  Common local arguments include:

\begin{itemize}
\item \cmdarg{-n $<$numprocs$>$}: Launch {\tt $<$numprocs$>$} number
  of copies of this executable.
  
\item \cmdarg{-arch $<$architecture$>$}: Launch the executable on
  nodes in the Open MPI universe that match this architecture.  An
  architecture is determined to be a match if the {\tt
    $<$architecture$>$} matches any subset of the GNU Autoconf
  architecture string on each of the target nodes (the \cmd{laminfo}
  command shows the GNU Autoconf configure string).
  
\item \cmdarg{$<$other arguments$>$}: When \cmd{mpiexec} first
  encounters an argument that it doesn't recognize, the remainder of
  the arguments will be passed back to \cmd{mpirun} to actually start
  the process.
\end{itemize}

The following example launches four copies of the
\cmd{my\_\-mpi\_\-program} executable in the Open MPI universe, using
default scheduling patterns:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpiexec -n 4 my_mpi_program
\end{lstlisting}
% Stupid emacs mode: $

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Launching MPMD Processes}

The ``\cmdarg{:}'' separator can be used to launch multiple
executables in the same MPI job.  Specifically, each process will
share a common \mpiconst{MPI\_\-COMM\_\-WORLD}.  For example, the
following launches a single \cmd{manager} process as well as a
\cmd{worker} process for every CPU in the Open MPI universe:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpiexec -n 1 manager : C worker
\end{lstlisting}
% Stupid emacs mode: $

Paired with the \cmd{-arch} flag, this can be especially helpful in
heterogeneous environments:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpiexec -arch solaris sol_program : -arch linux linux_program
\end{lstlisting}
% Stupid emacs mode: $

Even only ``slightly heterogeneous'' environments can run into
problems with shared libraries, different compilers, etc.  The
\cmd{-arch} flag can be used to differentiate between different
versions of the same operating system:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpiexec -arch solaris2.8 sol2.8_program : -arch solaris2.9 sol2.9_program
\end{lstlisting}
% Stupid emacs mode: $

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Launching MPI Processes with No Established Open MPI Universe}

The \cmd{-boot}, \cmd{-boot-args}, and \cmd{-machinefile} global
arguments can be used to launch the Open MPI RTE, run the MPI process(es),
and then take down the Open MPI RTE.  This conveniently wraps up several
Open MPI commands and provides ``one-shot'' execution of MPI processes.
For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpiexec -machinefile hostfile C my_mpi_program
\end{lstlisting}
% Stupid emacs mode: $

Some boot SSI modules do not require a hostfile; specifying the
\cmdarg{-boot} argument is sufficient in these cases:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpiexec -boot C my_mpi_program
\end{lstlisting}
% Stupid emacs mode: $

When \cmd{mpiexec} is used to boot the Open MPI RTE, it will do its best to
take down the Open MPI RTE even if errors occur, either during the boot
itself, or if an MPI process aborts (or the user hits Control-C).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{mpimsg} Command (Deprecated)}
\label{sec:commands-mpimsg}

The \cmd{mpimsg} command is deprecated.  It is only useful in a small
number of cases (specifically, when the \rpi{lamd} RPI module is
used), and may disappear in future Open MPI releases.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{mpirun} Command}
\label{sec:commands-mpirun}

The \cmd{mpirun} command is the main mechanism to launch MPI processes
in parallel.  

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Simple Examples}

Although \cmd{mpirun} supports many different modes of execution, most
users will likely only need to use a few of its capabilities.  It is
common to launch either one process per node or one process per CPU in
the Open MPI universe (CPU counts are established in the boot schema).  The
following two examples show these two cases:

\lstset{style=lam-cmdline}
\begin{lstlisting}
# Launch one copy of my_mpi_program on every schedulable node in the Open MPI universe
shell$ mpirun N my_mpi_program
\end{lstlisting}
% stupid emacs mode: $

\lstset{style=lam-cmdline}
\begin{lstlisting}
# Launch one copy of my_mpi_program on every schedulable CPU in the Open MPI universe
shell$ mpirun C my_mpi_program
\end{lstlisting}
% stupid emacs mode: $

The specific number of processes that are launched can be controlled
with the \cmdarg{-np} switch:

\lstset{style=lam-cmdline}
\begin{lstlisting}
# Launch four my_mpi_program processes
shell$ mpirun -np 4 my_mpi_program
\end{lstlisting}
% stupid emacs mode: $

The \cmdarg{-ssi} switch can be used to specify tunable parameters to
MPI processes.

\lstset{style=lam-cmdline}
\begin{lstlisting}
# Specify to use the usysv RPI module
shell$ mpirun -ssi rpi usysv C my_mpi_program
\end{lstlisting}
% stupid emacs mode: $

The available modules and their associated parameters are discussed in
detail in Chapter~\ref{sec:mpi-ssi}.

Arbitrary user arguments can also be passed to the user program.
\cmd{mpirun} will attempt to parse all options (looking for Open MPI
options) until it finds a \cmdarg{--}.  All arguments following
\cmdarg{--} are directly passed to the MPI application.

\lstset{style=lam-cmdline}
\begin{lstlisting}
# Pass three command line arguments to every instance of my_mpi_program
shell$ mpirun -ssi rpi usysv C my_mpi_program arg1 arg2 arg3
# Pass three command line arguments, escaped from parsing
shell$ mpirun -ssi rpi usysv C my_mpi_program -- arg1 arg2 arg3
\end{lstlisting}
% stupid emacs mode: $

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Controlling Where Processes Are Launched}

\cmd{mpirun} allows for fine-grained control of where to schedule
launched processes.  Note Open MPI uses the term ``schedule'' extensively
to indicate which nodes processes are launched on.  Open MPI does {\em not}
influence operating system semantics for prioritizing processes or
binding processes to specific CPUs.  The boot schema file can be used
to indicate how many CPUs are on a node, but this is only used for
scheduling purposes.  For a fuller description of CPU counts in boot
schemas, see Sections~\ref{sec:getting-started-hostfile}
and~\ref{sec:lam-ssi-boot-schema} on
pages~\pageref{sec:getting-started-hostfile}
and~\pageref{sec:lam-ssi-boot-schema}, respectively.

Open MPI offers two main scheduling nomenclatures: by node and by CPU.  For
example \cmdarg{N} means ``all schedulable nodes in the universe''
(``schedulable'' is defined in
Section~\ref{sec:commands-lamboot-no-schedule}).  Similarly,
\cmdarg{C} means ``all schedulable CPUs in the universe.''  

More fine-grained control is also possible -- nodes and CPUs can be
individually identified, or identified by ranges.  The syntax for
these concepts is \cmdarg{n$<$range$>$} and \cmdarg{c$<$range$>$},
respectively.  \cmdarg{$<$range$>$} can specify one or more elements
by listing integers separated by commas and dashes.  For example:

\begin{itemize}
\item \cmdarg{n3}: The node with an ID of 3.

\item \cmdarg{c2}: The CPU with an ID of 2.

\item \cmdarg{n2,4}: The nodes with IDs of 2 and 4.

\item \cmdarg{c2,4-7}: The CPUs with IDs of 2, 4, 5, 6, and 7.  Note
  that some of these CPUs may be on the same node(s).
\end{itemize}

Integers can range from 0 to the highest numbered node/CPU.  Note that
these nomenclatures can be mixed and matched on the \cmd{mpirun}
command line:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpirun n0 C manager-worker
\end{lstlisting}
% Stupid emacs mode: $

\noindent will launch the \cmd{manager-worker} program on \cmdarg{n0}
as well as on every schedulable CPU in the universe (yes, this means
that \cmdarg{n0} will likely be over-subscribed).

When running on SMP nodes, it is preferable to use the
\cmdarg{C}/\cmdarg{c$<$range$>$} nomenclature (with appropriate CPU
counts in the boot schema) to the \cmdarg{N}/\cmdarg{n$<$range$>$}
nomenclature because of how Open MPI will order ranks in \mcw.  For
example, consider a Open MPI universe of two four-way SMPs -- \cmdarg{n0}
and \cmdarg{n1} both have a CPU count of 4.  Using the following:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpirun C my_mpi_program
\end{lstlisting}
% Stupid emacs mode: $

\noindent will launch eight copies of \cmd{my\_\-mpi\_\-program}, four
on each node.  Open MPI will place as many adjoining \mcw\ ranks on the
same node as possible: \mcw\ ranks 0-3 will be scheduled on
\cmdarg{n0} and \mcw\ ranks 4-7 will be scheduled on \cmdarg{n1}.
Specifically, \cmdarg{C} schedules processes starting with \cmd{c0}
and incrementing the CPU index number.

Note that unless otherwise specified, Open MPI schedules processes by CPU
(vs.\ scheduling by node).  For example, using \cmd{mpirun}'s
\cmdarg{-np} switch to specify an absolute number of processes
schedules on a per-CPU basis.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Per-Process Controls}

\cmd{mpirun} allows for arbitrary, per-process controls such as
launching MPMD jobs, passing different command line arguments to
different \mcw\ ranks, etc.  This is accomplished by creating a text
file called an application schema that lists, one per line, the
location, relevant flags, user executable, and command line arguments
for each process.  For example (lines beginning with ``\#'' are
comments):

\lstset{style=lam-cmdline}
\begin{lstlisting}
# Start the manager on c0 with a specific set of command line options
c0 manager manager_arg1 manager_arg2 manager_arg3
# Start the workers on all available CPUs with different arguments
C worker worker_arg1 worker_arg2 worker_arg3
\end{lstlisting}

Note that the \cmdarg{-ssi} switch is {\em not} permissible in
application schema files; \cmdarg{-ssi} flags are considered to be
global to the entire MPI job, not specified per-process.  Application
schemas are described in more detail in the \file{appschema(5)} manual
page.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Ability to Pass Environment Variables}

All environment variables with names that begin with
\envvar{Open MPI\_\-MPI\_} are automatically passed to remote notes (unless
disabled via the \cmdarg{-nx} option to \cmd{mpirun}).  Additionally,
the \cmdarg{-x} option enables exporting of specific environment
variables to the remote nodes:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ Open MPI_MPI_FOO=``green eggs and ham''
shell$ export Open MPI_MPI_FOO
shell$ mpirun C -x DISPLAY,SEUSS=author samIam
\end{lstlisting}
% Stupid emacs mode: $

This will launch the \cmd{samIam} application on all available CPUs.
The \envvar{Open MPI\_\-MPI\_\-FOO}, \envvar{DISPLAY}, and \envvar{SEUSS}
environment variables will be created each the process environment
before the \cmd{smaIam} program is invoked.

Note that the parser for the \cmd{-x} option is currently not very
sophisticated -- it cannot even handle quoted values when defining new
environment variables.  Users are advised to set variables in the
environment prior to invoking \cmd{mpirun}, and only use \cmd{-x} to
export the variables to the remote nodes (not to define new
variables), if possible.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Current Working Directory Behavior}

Using the \cmd{-wd} option to \cmd{mpirun} allows specifying an
arbitrary working directory for the launched processes.  It can also
be used in application schema files to specify working directories on
specific nodes and/or for specific applications.

If the \cmdarg{-wd} option appears both in an application schema file
and on the command line, the schema file directory will override the
command line value.  \cmd{-wd} is mutually exclusive with \cmdarg{-D}.

If neither \cmdarg{-wd} nor \cmdarg{-D} are specified, the local node
will send the present working directory name from the \cmd{mpirun}
process to each of the remote nodes.  The remote nodes will then try
to change to that directory.  If they fail (e.g., if the directory
does not exist on that node), they will start from the user's home
directory.

All directory changing occurs before the user's program is invoked; it
does not wait until \mpifunc{MPI\_\-INIT} is called.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{mpitask} Command}
\label{sec:commands-mpitask}

The \cmd{mpitask} command shows a list of the processes running in the
Open MPI universe and a snapshot of their current MPI activity.  It is
usually invoked with no command line parameters, thereby showing
summary details of all processes currently running.  
%
Since \cmd{mpitask} only provides a snapshot view, it is not advisable
to use \cmd{mpitask} as a high-resolution debugger (see
Chapter~\ref{sec:debugging}, page~\pageref{sec:debugging}, for more
details on debugging MPI programs).  Instead, \cmd{mpitask} can be
used to provide answers to high-level questions such as ``Where is my
program hung?'' and ``Is my program making progress?''

The following example shows an MPI program running on four nodes,
sending a message of 524,288 integers around in a ring pattern.
Process 0 is running (i.e., not in an MPI function), while the other
three are blocked in \mpifunc{MPI\_\-RECV}.

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ mpitask
TASK (G/L)           FUNCTION      PEER|ROOT  TAG    COMM   COUNT   DATATYPE
0 ring               <running>
1/1 ring             Recv          0/0        201    WORLD  524288  INT
2/2 ring             Recv          1/1        201    WORLD  524288  INT
3/3 ring             Recv          2/2        201    WORLD  524288  INT
\end{lstlisting}
% Stupid emacs mode: $

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{recon} Command}
\label{sec:commands-recon}

The \cmd{recon} command is a quick test to see if the user's
environment is setup properly to boot the Open MPI RTE.  It takes most of
the same parameters as the \cmd{lamboot} command.

Although it does not boot the RTE, and does not definitively guarantee
that \cmd{lamboot} will succeed, it is a good tool for testing while
setting up first-time Open MPI users.  \cmd{recon} will display a
message when it has completed indicating whether it succeeded or
failed.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{tping} Command}
\label{sec:commands-tping}

The \cmd{tping} command can be used to verify the functionality of a
Open MPI universe.  It is used to send a ping message between the Open MPI
daemons that constitute the Open MPI RTE.  

It commonly takes two arguments: the set of nodes to ping (expressed
in \cmdarg{N} notation) and how many times to ping them.  Similar to the
Unix \cmd{ping} command, if the number of times to ping is not
specified, \cmd{tping} will continue until it is stopped (usually by
the user hitting Control-C).  The following example pings all nodes in
the Open MPI universe three times:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ tping N -c 3
  1 byte from 3 remote nodes and 1 local node: 0.002 secs
  1 byte from 3 remote nodes and 1 local node: 0.001 secs
  1 byte from 3 remote nodes and 1 local node: 0.001 secs

3 messages, 3 bytes (0.003K), 0.005 secs (1.250K/sec)
roundtrip min/avg/max: 0.001/0.002/0.002
\end{lstlisting}
% Stupid emacs mode: $

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{The \icmd{lamwipe} Command}
\label{sec:commands-lamwipe}

\changebegin{7.1}
The \cmd{lamwipe} command used to be called \idepcmd{wipe}.  The name
\idepcmd{wipe} has now been deprecated and although it still works in
this version of Open MPI, will be removed in future versions.  All
users are encouraged to start using \cmd{lamwipe} instead.
\changeend{7.1}

The \cmd{lamwipe} command is used as a ``last resort'' command, and is
typically only necessary if \cmd{lamhalt} fails.  This usually only
occurs in error conditions, such as if a node fails.  The
\cmd{lamwipe} command takes most of the same parameters as the
\cmd{lamboot} command -- it launches a process on each node in the
boot schema to kill the Open MPI RTE on that node.  Hence, it should be
used with the same (or an equivalent) boot schema file as was used
with \cmd{lamboot}.