% -*- latex -*- % % Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana % University Research and Technology % Corporation. All rights reserved. % Copyright (c) 2004-2005 The University of Tennessee and The University % of Tennessee Research Foundation. All rights % reserved. % Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, % University of Stuttgart. All rights reserved. % Copyright (c) 2004-2005 The Regents of the University of California. % All rights reserved. % $COPYRIGHT$ % % Additional copyrights may follow % % $HEADER$ % \chapter{Open MPI Command Quick Reference} \label{sec:commands} This section is intended to provide a quick reference of the major Open MPI commands. Each command also has its own manual page which typically provides more detail than this document. {\Huge JMS needs total overhaul} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamboot} Command} \label{sec:commands-lamboot} The \cmd{lamboot} command is used to start the Open MPI run-time environment (RTE). \cmd{lamboot} is typically the first command used before any other Open MPI command (notable exceptions are the wrapper compilers, which do not require the Open MPI RTE, and \cmd{mpiexec} which can launch its own Open MPI universe). \cmd{lamboot} can use any of the available \kind{boot} SSI modules; Section~\ref{sec:lam-ssi-boot} details the requirements and operations of each of the \kind{boot} SSI modules that are included in the Open MPI distribution. Common arguments that are used with the \cmd{lamboot} command are: \begin{itemize} \item \cmdarg{-b}: When used with the \boot{rsh} boot module, the ``fast'' boot algorithm is used which can noticeably speed up the execution time of \cmd{lamboot}. It can also be used where remote shell agents cannot provide output from remote nodes (e.g., in a Condor environment). Specifically, the ``fast'' algorithm assumes that the user's shell on the remote node is the same as the shell on the node where \cmd{lamboot} was invoked. \item \cmdarg{-d}: Print debugging output. This will print a {\em lot} of output, and is typically only necessary if \cmd{lamboot} fails for an unknown reason. The output is forwarded to standard out as well as either \file{/tmp} or syslog facilities. The amount of data produced can fill these filesystems, leading to general system problems. \item \cmdarg{-l}: Use local hostname resolution instead of centralized lookups. This is useful in environments where the same hostname may resolve to different IP addresses on different nodes (e.g., clusters based on Finite Neighborhood Networks\footnote{See \url{http://www.aggregate.org/} for more details.}). \changebegin{7.1} \item \cmdarg{-prefix $<$lam/install/path$>$}: Use the Open MPI installation specified in the $<$lam/install/path$>$ - where $<$lam/install/path$>$ is the top level directory where Open MPI is installed. This is typically used when a user has multiple Open MPI installations and want to switch between them without changing the dot files or PATH environment variable. This option is not compatible with Open MPI versions prior to 7.1. \changeend{7.1} \item \cmdarg{-s}: Close the \file{stdout} and \file{stderr} of the locally-launched Open MPI daemon (they are normally left open). This is necessary when invoking \cmd{lamboot} via a remote agent such as \cmd{rsh} or \cmd{ssh}. \item \cmdarg{-v}: Print verbose output. This is useful to show progress during \cmd{lamboot}'s progress. Unlike \cmdarg{-d}, \cmdarg{-v} does not forward output to a file or syslog. \item \cmdarg{-x}: Run the Open MPI RTE in fault-tolerant mode. \item \cmdarg{$<$filename$>$}: The name of the boot schema file. Boot schemas, while they can be as simple as a list of hostnames, can contain additional information and are discussed in detail in Sections~\ref{sec:getting-started-hostfile} and ~\ref{sec:lam-ssi-boot-schema}, pages~\pageref{sec:getting-started-hostfile} and~\pageref{sec:lam-ssi-boot-schema}, respectively. \end{itemize} Booting the Open MPI RTE is where most users (particularly first-time users) encounter problems. Each \kind{boot} module has its own specific requirements and prerequisites for success. Although \cmd{lamboot} typically prints detailed messages when errors occur, users are strongly encouraged to read Section~\ref{sec:lam-ssi-boot} for the details of the \kind{boot} module that they will be using. Additionally, the \cmdarg{-d} switch should be used to examine exactly what is happening to determine the actual source of the problem -- many problems with \cmd{lamboot} come from the operating system or the user's shell setup; not from within Open MPI itself. The most common \cmd{lamboot} example simply uses a hostfile to launch across an \cmd{rsh}/\cmd{ssh}-based cluster of nodes (the ``\cmdarg{-ssi boot rsh}'' is not technically necessary here, but it is specified to make this example correct in all environments): \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamboot -v -ssi boot rsh hostfile Open MPI 7.0/MPI 2 C++/ROMIO - Indiana University n0<1234> ssi:boot:base:linear: booting n0 (node1.cluster.example.com) n0<1234> ssi:boot:base:linear: booting n1 (node2.cluster.example.com) n0<1234> ssi:boot:base:linear: booting n2 (node3.cluster.example.com) n0<1234> ssi:boot:base:linear: booting n3 (node4.cluster.example.com) n0<1234> ssi:boot:base:linear: finished \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Multiple Sessions on the Same Node} In some cases (such as in batch-regulated environments), it is desirable to allow multiple universes owned by the same on the same node. The \ienvvar{TMPDIR}, \ienvvar{Open MPI\_\-MPI\_\-SESSION\_\-PREFIX}, and \ienvvar{Open MPI\_\-MPI\_\-SESSION\_\-SUFFIX} environment variables can be used to effect this behavior. The main issue is the location of Open MPI's session directory; each node in a Open MPI universe has a session directory in a well-known location in the filesystem that identifies how to contact the Open MPI daemon on that node. Multiple Open MPI universes can simultaneously co-exist on the same node as long as they have different session directories. Open MPI recognizes several batch environments and automatically adapts the session directory to be specific to a batch job. Hence, if the batch scheduler allocates multiple jobs from the same user to the same node, Open MPI will automatically do the ``right thing'' and ensure that the Open MPI universes from each job will not collide. % Sections~\ref{sec:misc-batch} and~\ref{sec:misc-session-directory} (starting on page~\pageref{sec:misc-batch}) discuss these issues in detail. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Avoiding Running on Specific Nodes} \label{sec:commands-lamboot-no-schedule} \index{no-schedule boot schema attribute@\cmd{no-schedule} boot schema attribute} Once the Open MPI universe is booted, processes can be launched on any node. The \cmd{mpirun}, \cmd{mpiexec}, and \cmd{lamexec} commands are most commonly used to launch jobs in the universe, and are typically used with the \cmdarg{N} and \cmdarg{C} nomenclatures (see the description of \cmd{mpirun} in Section~\ref{sec:commands-mpirun} for details on the \cmdarg{N} and \cmdarg{C} nomenclature) which launch jobs on all schedulable nodes and CPUs in the Open MPI universe, respectively. While finer-grained controls are available through \cmd{mpirun} (etc.), it can be convenient to simply mark some nodes as ``non-schedulable,'' and therefore avoid having \cmd{mpirun} (etc.) launch executables on those nodes when using \cmdarg{N} and \cmdarg{C} nomenclature. For example, it may be convenient to boot a Open MPI universe that includes a controller node (e.g., a desktop workstation) and a set of worker nodes. In this case, it is desirable to mark the desktop workstation as ``non-scheduable'' so that Open MPI will not launch executables there (by default). Consider the following boot schema: \lstset{style=lam-shell} \begin{lstlisting} # Mark my_workstation as ``non-schedulable'' my_workstation.office.example.com schedule=no # All the other nodes are, by default, schedulable node1.cluster.example.com node2.cluster.example.com node3.cluster.example.com node4.cluster.example.com \end{lstlisting} Booting with this schema allows the convenienve of: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpirun C my_mpi_program \end{lstlisting} % stupid emacs mode: $ \noindent which will only run \cmd{my\_\-mpi\_\-program} on the four cluster nodes (i.e., not the workstation). % Note that this behavior {\em only} applies to the \cmdarg{C} and \cmdarg{N} designations; Open MPI will always allow execution on any node when using the \cmdarg{nX} or \cmdarg{cX} notation: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpirun c0 C my_mpi_program \end{lstlisting} % stupid emacs mode: $ \noindent which will run \cmd{my\_\-mpi\_\-program} on all five nodes in the Open MPI universe. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamcheckpoint} Command} \label{sec:commands-lamcheckpoint} \changebegin{7.1} The \cmd{lamcheckpoint} command is provided to checkpoint a MPI application. One of the arguments to \cmd{lamcheckpoint} is the name of the checkpoint/restart module (which can be either one of \crssi{blcr} and \crssi{self}). Additional arguments to \cmd{lamcheckpoint} depend of the selected checkpoint/restart module. The name of the module can be specified by passing the \crssi{cr} SSI parameter. Common arguments that are used with the \cmd{lamcheckpoint} command are: \begin{itemize} \item \cmdarg{-ssi}: Just like with \cmd{mpirun}, the \cmdarg{-ssi} flag can be used to pass key=value pairs to Open MPI. Indeed, it is required to pass at least one SSI parameter: \ssiparam{cr}, indicating which \kind{cr} module to use for checkpointing. \item \cmdarg{-pid}: Indicate the PID of \cmd{mpirun} to checkpoint. \end{itemize} \noindent Notes: \begin{itemize} \item If the \crssi{blcr} \kind{cr} module is selected, the name of the directory for storing the checkpoint files and the PID of \cmd{mpirun} should be passed as SSI parameters to \cmd{lamcheckpoint}. \item If the \crssi{self} \kind{cr} module is selected, the PID of \cmd{mpirun} should be passed via the \cmdarg{-pid} parameter. \end{itemize} \changeend{7.1} See Section~\ref{sec:mpi-ssi-cr} for more detail about the checkpoint/restart capabilities of Open MPI, including details about the \crssi{blcr} and \crssi{self} \kind{cr} modules. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamclean} Command} \label{sec:commands-lamclean} The \cmd{lamclean} command is provided to clean up the Open MPI universe. It is typically only necessary when MPI processes terminate ``badly,'' and potentially leave resources allocated in the Open MPI universe (such as MPI-2 published names, processes, or shared memory). The \cmd{lamclean} command will kill {\em all} processes running in the Open MPI universe, and free {\em all} resources that were associated with them (including unpublishing MPI-2 dynamicly published names). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamexec} Command} \label{sec:commands-lamexec} The \cmd{lamexec} command is similar to \cmd{mpirun} but is used for non-MPI programs. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamexec N uptime 5:37pm up 21 days, 23:49, 5 users, load average: 0.31, 0.26, 0.25 5:37pm up 21 days, 23:49, 2 users, load average: 0.01, 0.00, 0.00 5:37pm up 21 days, 23:50, 3 users, load average: 0.01, 0.00, 0.00 5:37pm up 21 days, 23:50, 2 users, load average: 0.87, 0.81, 0.80 \end{lstlisting} % Stupid emacs: $ Most of the parameters and options that are available to \cmd{mpirun} are also available to \cmd{lamexec}. See the \cmd{mpirun} description in Section~\ref{sec:commands-mpirun} for more details. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamgrow} Command} \label{sec:commands-lamgrow} The \cmd{lamgrow} command adds a single node to the Open MPI universe. It must use the same \kind{boot} module that was used to initially boot the Open MPI universe. \cmd{lamgrow} must be run from a node already in the Open MPI universe. Common parameters include: \begin{itemize} \item \cmdarg{-v}: Verbose mode. \item \cmdarg{-d}: Debug mode; enables a {\em lot} of diagnostic output. \item \cmdarg{-n $<$nodeid$>$}: Assign the new host the node ID \cmdarg{nodeid}. \cmdarg{nodeid} must be an unused node ID. If \cmdarg{-n} is not specified, Open MPI will find the lowest node ID that is not being used. \item \cmdarg{-no-schedule}: Has the same effect as putting ``{\tt no\_\-schedule=yes}'' in the boot schema. This means that the \cmdarg{C} and \cmdarg{N} expansion used in \cmd{mpirun} and \cmd{lamexec} will not include this node. \item \cmdarg{-ssi $<$key$>$ $<$value$>$}: Pass in SSI parameter \cmdarg{key} with the value \cmdarg{value}. \item \cmdarg{$<$hostname$>$}: The name of the host to expand the universe to. \end{itemize} For example, the following adds the node \host{blinky} to the existing Open MPI universe using the \boot{rsh} boot module: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamgrow -ssi boot rsh blinky.cluster.example.com \end{lstlisting} % Stupid emacs: $ Note that \cmd{lamgrow} cannot grow a Open MPI universe that only contains one node that has an IP address of 127.0.0.1 (e.g., if \cmd{lamboot} was run with the default boot schema that only contains the name \host{localhost}). In this case, \cmd{lamgrow} will print an error and abort without adding the new node. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamhalt} Command} \label{sec:commands-lamhalt} The \cmd{lamhalt} command is used to shut down the Open MPI RTE. Typically, \cmd{lamhalt} can simply be run with no command line parameters and it will shut down the Open MPI RTE. Optionally, the \cmdarg{-v} or \cmdarg{-d} arguments can be used to make \cmd{lamhalt} be verbose or extremely verbose, respectively. There are a small number of cases where \cmd{lamhalt} will fail. For example, if a Open MPI daemon becomes unresponsive (e.g., the daemon was killed), \cmd{lamhalt} may fail to shut down the entire Open MPI universe. It will eventually timeout and therefore complete in finite time, but you may want to use the last-resort \cmd{lamwipe} command (see Section~\ref{sec:commands-lamwipe}). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{laminfo} Command} \label{sec:commands-laminfo} The \cmd{laminfo} command can be used to query the capabilities of the Open MPI installation. Running \cmd{laminfo} with no parameters shows a prettyprint summary of information. Using the \cmdarg{-parsable} command line switch shows the same summary information, but in a format that should be relatively easy to parse with common unix tools such as \cmd{grep}, \cmd{cut}, \cmd{awk}, etc. \cmd{laminfo} supports a variety of command line options to query for specific information. The \cmdarg{-h} option shows a complete listing of all options. Some of the most common options include: \begin{itemize} \item \cmdarg{-arch}: Show the architecture that Open MPI was configured for. \item \cmdarg{-path}: Paired with a second argument, display various paths relevant to the Open MPI installation. Valid second arguments include: \begin{itemize} \item \cmdarg{prefix}: Main installation prefix \item \cmdarg{bindir}: Where the Open MPI executables are located \item \cmdarg{libdir}: Where the Open MPI libraries are located \item \cmdarg{incdir}: Where the Open MPI include files are located \item \cmdarg{pkglibdir}: Where dynamic SSI modules are installed\footnote{Dynamic SSI modules are not supported in Open MPI 7.0, but will be supported in future versions.} \item \cmdarg{sysconfdir}: Where the Open MPI help files are located \end{itemize} \item \cmdarg{-version}: Paired with two addition options, display the version of either Open MPI or one or more SSI modules. The first argument identifies what to report the version of, and can be any of the following: \begin{itemize} \item \cmdarg{lam}: Version of Open MPI \item \cmdarg{boot}: Version of all boot modules \item \cmdarg{boot:module}: Version of a specific boot module \item \cmdarg{coll}: Version of all coll modules \item \cmdarg{coll:module}: Version of a specific coll module \item \cmdarg{cr}: Version of all cr modules \item \cmdarg{cr:module}: Version of a specific cr module \item \cmdarg{rpi}: Version of all rpi modules \item \cmdarg{rpi:module}: Version of a specific rpi module \end{itemize} The second argument specifies the scope of the version number to display -- whether to show the entire version number string, or just one component of it: \begin{itemize} \item \cmdarg{full}: Display the entire version number string \item \cmdarg{major}: Display the major version number \item \cmdarg{minor}: Display the minor version number \item \cmdarg{release}: Display the release version number \item \cmdarg{alpha}: Display the alpha version number \item \cmdarg{beta}: Display the beta version number \item \cmdarg{svn}: Display the SVN version number\footnote{The value will either be 0 (not built from SVN), 1 (built from a Subverstion checkout) or a date encoded in the form YYYYMMDD (built from a nightly tarball on the given date)} \end{itemize} \changebegin{7.1} \item \cmdarg{-param}: Paired with two additional arguments, display the SSI parameters for a given type and/or module. The first argument can be any of the valid SSI types or the special name ``base,'' indicating the SSI framework itself. The second argument can be any valid module name. Additionally, either argument can be the wildcard ``any'' which will match any valid SSI type and/or module. \changeend{7.1} \end{itemize} Multiple options can be combined to query several attributes at once: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ laminfo -parsable -arch -version lam major -version rpi:tcp full -param rpi tcp version:lam:7 ssi:boot:rsh:version:ssi:1.0 ssi:boot:rsh:version:api:1.0 ssi:boot:rsh:version:module:7.0 arch:i686-pc-linux-gnu ssi:rpi:tcp:param:rpi_tcp_short:65536 ssi:rpi:tcp:param:rpi_tcp_sockbuf:-1 ssi:rpi:tcp:param:rpi_tcp_priority:20 \end{lstlisting} % Stupid emacs: $ Note that three version numbers are returned for the \rpi{tcp} module. The first (\cmdarg{ssi}) indicates the overall SSI version that the module conforms to, the second (\cmdarg{api}) indicates what version of the \kind{rpi} API the module conforms to, and the last (\cmdarg{module}) indicates the version of the module itself. Running \cmd{laminfo} with no arguments provides a wealth of information about your Open MPI installation (we ask for this output when reporting problems to the Open MPI general user's mailing list -- see Section \ref{troubleshooting:mailing-lists} on page \pageref{troubleshooting:mailing-lists}). Most of the output fields are self-explanitory; two that are worth explaining are: \begin{itemize} \item Debug support: This indicates whether your Open MPI installation was configured with the \confflag{with-debug} option. It is generally only used by the Open MPI Team for development and maintenance of Open MPI itself; it does {\em not} indicate whether user's MPI applications can be debugged (specifically: user's MPI applications can {\em always} be debugged, regardless of this setting). This option defaults to ``no''; users are discouraged from using this option. See the Install Guide for more information about \confflag{with-debug}. \item Purify clean: This indicates whether your Open MPI installation was configured with the \confflag{with-purify} option. This option is necessary to prevent a number of false positives when using memory-checking debuggers such as Purify, Valgrind, and bcheck. It is off by default because it can cause slight performance degredation in MPI applications. See the Install Guide for more information about \confflag{with-purify}. \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamnodes} Command} \label{sec:commands-lamnodes} Open MPI was specifically designed to abstract away hostnames once \cmd{lamboot} has completed successfully. However, for various reasons (usually related to system-administration concerns, and/or for creating human-readable reports), it can be desirable to retrieve the hostnames of Open MPI nodes long after \icmd{lamboot}. The command \cmd{lamnodes} can be used for this purpose. It accepts both the \cmdarg{N} and \cmdarg{C} syntax from \cmd{mpirun}, and will return the corresponding names of the specified nodes. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamnodes N \end{lstlisting} % Stupid emacs: $ \noindent will return the node that each CPU is located on, the hostname of that node, the total number of CPUs on each, and any flags that are set on that node. Specific nodes can also be queried: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamnodes n0,3 \end{lstlisting} % Stupid emacs: $ \noindent will return the node, hostname, number of CPUs, and flags on n0 and n3. Command line arguments can be used to customize the output of \cmd{lamnodes}. These include: \begin{itemize} \item \cmdarg{-c}: Suppress printing CPU counts \item \cmdarg{-i}: Print IP addresses instead of IP names \item \cmdarg{-n}: Suppress printing Open MPI node IDs \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamrestart} Command} The \cmd{lamrestart} can be used to restart a previously-checkpointed MPI application. The arguments to \cmd{lamrestart} depend on the selected checkpoint/restart module. Regardless of the checkpoint/restart module used, invoking \cmd{lamrestart} results in a new \cmd{mpirun} being launched. The SSI parameter \ssiparam{cr} must be used to specify which checkpoint/restart module should be used to restart the application. Currently, only two values are possible: \ssiparam{blcr} and \ssiparam{self}. \begin{itemize} \item If the \crssi{blcr} module is selected, the SSI parameter \issiparam{cr\_\-blcr\_\-context\_\-file} should be used to pass in the filename of the context file that was created during a pevious successful checkpoint. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamrestart -ssi cr blcr -ssi cr_blcr_context_file filename \end{lstlisting} % Stupid emacs: $ \item If the \crssi{self} module is selected, the SSI parameter \issiparam{cr\_\-restart\_\-args} must be passed with the arguments to be passed to \cmd{mpirun} to restart the application. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamrestart -ssi cr self -ssi cr_restart_args "args to mpirun" \end{lstlisting} % Stupid emacs: $ \end{itemize} See Section~\ref{sec:mpi-ssi-cr} for more detail about the checkpoint/restart capabilities of Open MPI, including details about the \crssi{blcr} and \crssi{self} \kind{cr} modules. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamshrink} Command} \label{sec:commands-lamshrink} The \cmd{lamshrink} command is used to remove a node from a Open MPI universe: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamshrink n3 \end{lstlisting} % Stupid emacs: $ \noindent removes node n3 from the Open MPI universe. Note that all nodes with ID's greater than 3 will not have their ID's reduced by one -- n3 simply becomes an empty slot in the Open MPI universe. \cmd{mpirun} and \cmd{lamexec} will still function correctly, even when used with \cmdarg{C} and \cmdarg{N} notation -- they will simply skip the n3 since there is no longer an operational node in that slot. Note that the \cmd{lamgrow} command can optionally be used to fill the empty slot with a new node. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{mpicc}, \icmd{mpiCC} / \icmd{mpic++}, and \icmd{mpif77} Commands} \label{sec:commands-wrappers} \index{wrapper compilers} Compiling MPI applications can be a complicated process because the list of compiler and linker flags required to successfully compile and link a Open MPI application not only can be quite long, it can change depending on the particular configuration that Open MPI was installed with. For example, if Open MPI includes native support for Myrinet hardware, the \cmdarg{-lgm} flag needs to be used when linking MPI executables. To hide all this complexity, ``wrapper'' compilers are provided that handle all of this automatically. They are called ``wrapper'' compilers because all they do is add relevant compiler and linker flags to the command line before invoking the real back-end compiler to actually perform the compile/link. Most command line arugments are passed straight through to the back-end compiler without modification. Therefore, to compile an MPI application, use the wrapper compilers exactly as you would use the real compiler. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc -O -c main.c shell$ mpicc -O -c foo.c shell$ mpicc -O -c bar.c shell$ mpicc -O -o main main.o foo.o bar.o \end{lstlisting} This compiles three C source code files and links them together into a single executable. No additional \cmdarg{-I}, \cmdarg{-L}, or \cmdarg{-l} arguments are required. The main exceptions to what flags are not passed through to the back-end compiler are: \begin{itemize} \item \cmdarg{-showme}: Used to show what the wrapper compiler would have executed. This is useful to see the full compile/link line would have been executed. For example (your output may differ from what is shown below, depending on your installed Open MPI configuration): \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc -O -c main.c -showme gcc -I/usr/local/lam/include -pthread -O -c foo.c \end{lstlisting} % Stupid emacs mode: $ \lstset{style=lam-cmdline} \begin{lstlisting} # The output line shown below is word wrapped in order to fit nicely in the document margins shell$ mpicc -O -o main main.o foo.o bar.o -showme gcc -I/usr/local/lam/include -pthread -O -o main main.o foo.o bar.o \ -L/usr/local/lam/lib -llammpio -lpmpi -llamf77mpi -lmpi -llam -lutil \ -pthread \end{lstlisting} % Stupid emacs mode: $ \changebegin{7.1} Two notable sub-flags are: \begin{itemize} \item \cmdarg{-showme:compile}: Show only the compile flags, suitable for substitution into \envvar{CFLAGS}. \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc -O -c main.c -showme:compile -I/usr/local/lam/include -pthread \end{lstlisting} % Stupid emacs mode: $ \item \cmdarg{-showme:link}: Show only the linker flags (which are actually \envvar{LDFLAGS} and \envvar{LIBS} mixed together), suitable for substitution into \envvar{LIBS}. \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc -O -o main main.o foo.o bar.o -showme:link -L/usr/local/lam/lib -llammpio -lpmpi -llamf77mpi -lmpi -llam -lutil -pthread \end{lstlisting} % Stupid emacs mode: $ \end{itemize} \changeend{7.1} \item \cmdarg{-lpmpi}: When compiling a user MPI application, the \cmdarg{-lpmpi} argument is used to indicate that MPI profiling support should be included. The wrapper compiler may alter the exact placement of this argument to ensure that proper linker dependency semantics are preserved. \end{itemize} \changebegin{7.1} Neither the compiler nor linker flags can be overridden at run-time. The back-end compiler, however, can be. Environment variables can be used for this purpose: \begin{itemize} \item \ienvvar{Open MPIMPICC} (deprecated name: \idepenvvar{Open MPIHCC}): Overrides the default C compiler in the \cmd{mpicc} wrapper compiler. \item \ienvvar{Open MPIMPICXX} (deprecated name: \idepenvvar{Open MPIHCP}): Overrides the default C compiler in the \cmd{mpicc} wrapper compiler. \item \ienvvar{Open MPIMPIF77} (deprecated name: \idepenvvar{Open MPIHF77}): Overrides the default C compiler in the \cmd{mpicc} wrapper compiler. \end{itemize} For example (for Bourne-like shells): \lstset{style=lam-cmdline} \begin{lstlisting} shell$ Open MPIPICC=cc shell$ export Open MPIMPICC shell$ mpicc my_application.c -o my_application \end{lstlisting} % Stupid emacs mode: $ For csh-like shells: \lstset{style=lam-cmdline} \begin{lstlisting} shell% setenv Open MPIPICC cc shell% mpicc my_application.c -o my_application \end{lstlisting} All this being said, it is {\em strongly} recommended to use the wrapper compilers -- and their default underlying compilers -- for all compiling and linking of MPI applications. Strange behavior can occur in MPI applications if Open MPI was configured and compiled with one compiler and then user applications were compiled with a different underlying compiler, to include: failure to compile, failure to link, seg faults and other random bad behavior at run-time. Finally, note that the wrapper compilers only add all the Open MPI-specific flags when a command-line argument that does not begin with a dash (``-'') is present. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc gcc: no input files shell$ mpicc --version gcc (GCC) 3.2.2 (Mandrake Linux 9.1 3.2.2-3mdk) Copyright (C) 2002 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. \end{lstlisting} \changeend{7.1} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Deprecated Names} Previous versions of Open MPI used the names \idepcmd{hcc}, \idepcmd{hcp}, and \idepcmd{hf77} for the wrapper compilers. While these command names still work (they are simply symbolic links to the real wrapper compilers \cmd{mpicc}, \cmd{mpiCC}/\cmd{mpic++}, and \cmd{mpif77}, respectively), their use is deprecated. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{mpiexec} Command} \label{sec:commands-mpiexec} The \cmd{mpiexec} command is used to launch MPI programs. It is similar to, but slightly different than, \cmd{mpirun}.\footnote{The reason that there are two methods to launch MPI executables is because the MPI-2 standard suggests the use of \cmd{mpiexec} and provides standardized command line arguments. Hence, even though Open MPI already had an \cmd{mpirun} command to launch MPI executables, \cmd{mpiexec} was added to comply with the standard.} Although \cmd{mpiexec} is simply a wrapper around other Open MPI commands (including \cmd{lamboot}, \cmd{mpirun}, and \cmd{lamhalt}), it ties their functionality together and provides a unified interface for launching MPI processes. % Specifically, \cmd{mpiexec} offers two features from command line flags that require multiple steps when using other Open MPI commands: launching MPMD MPI processes and launching MPI processes when there is no existing Open MPI universe. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{General Syntax} The general form of \cmd{mpiexec} commands is: \lstset{style=lam-cmdline} \begin{lstlisting} mpiexec [global_args] local_args1 [: local_args2 [...]] \end{lstlisting} Global arguments are applied to all MPI processes that are launched. They must be specified before any local arguments. Common global arguments include: \begin{itemize} \item \cmdarg{-boot}: Boot the Open MPI RTE before launching the MPI processes. \item \cmdarg{-boot-args $<$args$>$}: Pass \cmdarg{$<$args$>$} to the back-end \cmd{lamboot}. Implies \cmdarg{-boot}. \item \cmdarg{-machinefile $<$filename$>$}: Specify {\tt $<$filename$>$} as the boot schema to use when invoking the back-end \cmd{lamboot}. Implies \cmdarg{-boot}. \changebegin{7.1} \item \cmdarg{-prefix $<$lam/install/path$>$}: Use the Open MPI installation specified in the $<$lam/install/path$>$ - where $<$lam/install/path$>$ is the top level directory where Open MPI is ``installed''. This is typically used when a user has multiple Open MPI installations and want to switch between them without changing the dot files or PATH environment variable. This option is not compatible with Open MPI versions prior to 7.1. \changeend{7.1} \item \cmdarg{-ssi $<$key$>$ $<$value$>$}: Pass the SSI {\tt $<$key$>$} and {\tt $<$value$>$} arguments to the back-end \cmd{mpirun} command. \end{itemize} Local arguments are specific to an individual MPI process that will be launched. They are specified along with the executable that will be launched. Common local arguments include: \begin{itemize} \item \cmdarg{-n $<$numprocs$>$}: Launch {\tt $<$numprocs$>$} number of copies of this executable. \item \cmdarg{-arch $<$architecture$>$}: Launch the executable on nodes in the Open MPI universe that match this architecture. An architecture is determined to be a match if the {\tt $<$architecture$>$} matches any subset of the GNU Autoconf architecture string on each of the target nodes (the \cmd{laminfo} command shows the GNU Autoconf configure string). \item \cmdarg{$<$other arguments$>$}: When \cmd{mpiexec} first encounters an argument that it doesn't recognize, the remainder of the arguments will be passed back to \cmd{mpirun} to actually start the process. \end{itemize} The following example launches four copies of the \cmd{my\_\-mpi\_\-program} executable in the Open MPI universe, using default scheduling patterns: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -n 4 my_mpi_program \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Launching MPMD Processes} The ``\cmdarg{:}'' separator can be used to launch multiple executables in the same MPI job. Specifically, each process will share a common \mpiconst{MPI\_\-COMM\_\-WORLD}. For example, the following launches a single \cmd{manager} process as well as a \cmd{worker} process for every CPU in the Open MPI universe: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -n 1 manager : C worker \end{lstlisting} % Stupid emacs mode: $ Paired with the \cmd{-arch} flag, this can be especially helpful in heterogeneous environments: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -arch solaris sol_program : -arch linux linux_program \end{lstlisting} % Stupid emacs mode: $ Even only ``slightly heterogeneous'' environments can run into problems with shared libraries, different compilers, etc. The \cmd{-arch} flag can be used to differentiate between different versions of the same operating system: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -arch solaris2.8 sol2.8_program : -arch solaris2.9 sol2.9_program \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Launching MPI Processes with No Established Open MPI Universe} The \cmd{-boot}, \cmd{-boot-args}, and \cmd{-machinefile} global arguments can be used to launch the Open MPI RTE, run the MPI process(es), and then take down the Open MPI RTE. This conveniently wraps up several Open MPI commands and provides ``one-shot'' execution of MPI processes. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -machinefile hostfile C my_mpi_program \end{lstlisting} % Stupid emacs mode: $ Some boot SSI modules do not require a hostfile; specifying the \cmdarg{-boot} argument is sufficient in these cases: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -boot C my_mpi_program \end{lstlisting} % Stupid emacs mode: $ When \cmd{mpiexec} is used to boot the Open MPI RTE, it will do its best to take down the Open MPI RTE even if errors occur, either during the boot itself, or if an MPI process aborts (or the user hits Control-C). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{mpimsg} Command (Deprecated)} \label{sec:commands-mpimsg} The \cmd{mpimsg} command is deprecated. It is only useful in a small number of cases (specifically, when the \rpi{lamd} RPI module is used), and may disappear in future Open MPI releases. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{mpirun} Command} \label{sec:commands-mpirun} The \cmd{mpirun} command is the main mechanism to launch MPI processes in parallel. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Simple Examples} Although \cmd{mpirun} supports many different modes of execution, most users will likely only need to use a few of its capabilities. It is common to launch either one process per node or one process per CPU in the Open MPI universe (CPU counts are established in the boot schema). The following two examples show these two cases: \lstset{style=lam-cmdline} \begin{lstlisting} # Launch one copy of my_mpi_program on every schedulable node in the Open MPI universe shell$ mpirun N my_mpi_program \end{lstlisting} % stupid emacs mode: $ \lstset{style=lam-cmdline} \begin{lstlisting} # Launch one copy of my_mpi_program on every schedulable CPU in the Open MPI universe shell$ mpirun C my_mpi_program \end{lstlisting} % stupid emacs mode: $ The specific number of processes that are launched can be controlled with the \cmdarg{-np} switch: \lstset{style=lam-cmdline} \begin{lstlisting} # Launch four my_mpi_program processes shell$ mpirun -np 4 my_mpi_program \end{lstlisting} % stupid emacs mode: $ The \cmdarg{-ssi} switch can be used to specify tunable parameters to MPI processes. \lstset{style=lam-cmdline} \begin{lstlisting} # Specify to use the usysv RPI module shell$ mpirun -ssi rpi usysv C my_mpi_program \end{lstlisting} % stupid emacs mode: $ The available modules and their associated parameters are discussed in detail in Chapter~\ref{sec:mpi-ssi}. Arbitrary user arguments can also be passed to the user program. \cmd{mpirun} will attempt to parse all options (looking for Open MPI options) until it finds a \cmdarg{--}. All arguments following \cmdarg{--} are directly passed to the MPI application. \lstset{style=lam-cmdline} \begin{lstlisting} # Pass three command line arguments to every instance of my_mpi_program shell$ mpirun -ssi rpi usysv C my_mpi_program arg1 arg2 arg3 # Pass three command line arguments, escaped from parsing shell$ mpirun -ssi rpi usysv C my_mpi_program -- arg1 arg2 arg3 \end{lstlisting} % stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Controlling Where Processes Are Launched} \cmd{mpirun} allows for fine-grained control of where to schedule launched processes. Note Open MPI uses the term ``schedule'' extensively to indicate which nodes processes are launched on. Open MPI does {\em not} influence operating system semantics for prioritizing processes or binding processes to specific CPUs. The boot schema file can be used to indicate how many CPUs are on a node, but this is only used for scheduling purposes. For a fuller description of CPU counts in boot schemas, see Sections~\ref{sec:getting-started-hostfile} and~\ref{sec:lam-ssi-boot-schema} on pages~\pageref{sec:getting-started-hostfile} and~\pageref{sec:lam-ssi-boot-schema}, respectively. Open MPI offers two main scheduling nomenclatures: by node and by CPU. For example \cmdarg{N} means ``all schedulable nodes in the universe'' (``schedulable'' is defined in Section~\ref{sec:commands-lamboot-no-schedule}). Similarly, \cmdarg{C} means ``all schedulable CPUs in the universe.'' More fine-grained control is also possible -- nodes and CPUs can be individually identified, or identified by ranges. The syntax for these concepts is \cmdarg{n$<$range$>$} and \cmdarg{c$<$range$>$}, respectively. \cmdarg{$<$range$>$} can specify one or more elements by listing integers separated by commas and dashes. For example: \begin{itemize} \item \cmdarg{n3}: The node with an ID of 3. \item \cmdarg{c2}: The CPU with an ID of 2. \item \cmdarg{n2,4}: The nodes with IDs of 2 and 4. \item \cmdarg{c2,4-7}: The CPUs with IDs of 2, 4, 5, 6, and 7. Note that some of these CPUs may be on the same node(s). \end{itemize} Integers can range from 0 to the highest numbered node/CPU. Note that these nomenclatures can be mixed and matched on the \cmd{mpirun} command line: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpirun n0 C manager-worker \end{lstlisting} % Stupid emacs mode: $ \noindent will launch the \cmd{manager-worker} program on \cmdarg{n0} as well as on every schedulable CPU in the universe (yes, this means that \cmdarg{n0} will likely be over-subscribed). When running on SMP nodes, it is preferable to use the \cmdarg{C}/\cmdarg{c$<$range$>$} nomenclature (with appropriate CPU counts in the boot schema) to the \cmdarg{N}/\cmdarg{n$<$range$>$} nomenclature because of how Open MPI will order ranks in \mcw. For example, consider a Open MPI universe of two four-way SMPs -- \cmdarg{n0} and \cmdarg{n1} both have a CPU count of 4. Using the following: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpirun C my_mpi_program \end{lstlisting} % Stupid emacs mode: $ \noindent will launch eight copies of \cmd{my\_\-mpi\_\-program}, four on each node. Open MPI will place as many adjoining \mcw\ ranks on the same node as possible: \mcw\ ranks 0-3 will be scheduled on \cmdarg{n0} and \mcw\ ranks 4-7 will be scheduled on \cmdarg{n1}. Specifically, \cmdarg{C} schedules processes starting with \cmd{c0} and incrementing the CPU index number. Note that unless otherwise specified, Open MPI schedules processes by CPU (vs.\ scheduling by node). For example, using \cmd{mpirun}'s \cmdarg{-np} switch to specify an absolute number of processes schedules on a per-CPU basis. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Per-Process Controls} \cmd{mpirun} allows for arbitrary, per-process controls such as launching MPMD jobs, passing different command line arguments to different \mcw\ ranks, etc. This is accomplished by creating a text file called an application schema that lists, one per line, the location, relevant flags, user executable, and command line arguments for each process. For example (lines beginning with ``\#'' are comments): \lstset{style=lam-cmdline} \begin{lstlisting} # Start the manager on c0 with a specific set of command line options c0 manager manager_arg1 manager_arg2 manager_arg3 # Start the workers on all available CPUs with different arguments C worker worker_arg1 worker_arg2 worker_arg3 \end{lstlisting} Note that the \cmdarg{-ssi} switch is {\em not} permissible in application schema files; \cmdarg{-ssi} flags are considered to be global to the entire MPI job, not specified per-process. Application schemas are described in more detail in the \file{appschema(5)} manual page. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Ability to Pass Environment Variables} All environment variables with names that begin with \envvar{Open MPI\_\-MPI\_} are automatically passed to remote notes (unless disabled via the \cmdarg{-nx} option to \cmd{mpirun}). Additionally, the \cmdarg{-x} option enables exporting of specific environment variables to the remote nodes: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ Open MPI_MPI_FOO=``green eggs and ham'' shell$ export Open MPI_MPI_FOO shell$ mpirun C -x DISPLAY,SEUSS=author samIam \end{lstlisting} % Stupid emacs mode: $ This will launch the \cmd{samIam} application on all available CPUs. The \envvar{Open MPI\_\-MPI\_\-FOO}, \envvar{DISPLAY}, and \envvar{SEUSS} environment variables will be created each the process environment before the \cmd{smaIam} program is invoked. Note that the parser for the \cmd{-x} option is currently not very sophisticated -- it cannot even handle quoted values when defining new environment variables. Users are advised to set variables in the environment prior to invoking \cmd{mpirun}, and only use \cmd{-x} to export the variables to the remote nodes (not to define new variables), if possible. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Current Working Directory Behavior} Using the \cmd{-wd} option to \cmd{mpirun} allows specifying an arbitrary working directory for the launched processes. It can also be used in application schema files to specify working directories on specific nodes and/or for specific applications. If the \cmdarg{-wd} option appears both in an application schema file and on the command line, the schema file directory will override the command line value. \cmd{-wd} is mutually exclusive with \cmdarg{-D}. If neither \cmdarg{-wd} nor \cmdarg{-D} are specified, the local node will send the present working directory name from the \cmd{mpirun} process to each of the remote nodes. The remote nodes will then try to change to that directory. If they fail (e.g., if the directory does not exist on that node), they will start from the user's home directory. All directory changing occurs before the user's program is invoked; it does not wait until \mpifunc{MPI\_\-INIT} is called. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{mpitask} Command} \label{sec:commands-mpitask} The \cmd{mpitask} command shows a list of the processes running in the Open MPI universe and a snapshot of their current MPI activity. It is usually invoked with no command line parameters, thereby showing summary details of all processes currently running. % Since \cmd{mpitask} only provides a snapshot view, it is not advisable to use \cmd{mpitask} as a high-resolution debugger (see Chapter~\ref{sec:debugging}, page~\pageref{sec:debugging}, for more details on debugging MPI programs). Instead, \cmd{mpitask} can be used to provide answers to high-level questions such as ``Where is my program hung?'' and ``Is my program making progress?'' The following example shows an MPI program running on four nodes, sending a message of 524,288 integers around in a ring pattern. Process 0 is running (i.e., not in an MPI function), while the other three are blocked in \mpifunc{MPI\_\-RECV}. \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpitask TASK (G/L) FUNCTION PEER|ROOT TAG COMM COUNT DATATYPE 0 ring 1/1 ring Recv 0/0 201 WORLD 524288 INT 2/2 ring Recv 1/1 201 WORLD 524288 INT 3/3 ring Recv 2/2 201 WORLD 524288 INT \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{recon} Command} \label{sec:commands-recon} The \cmd{recon} command is a quick test to see if the user's environment is setup properly to boot the Open MPI RTE. It takes most of the same parameters as the \cmd{lamboot} command. Although it does not boot the RTE, and does not definitively guarantee that \cmd{lamboot} will succeed, it is a good tool for testing while setting up first-time Open MPI users. \cmd{recon} will display a message when it has completed indicating whether it succeeded or failed. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{tping} Command} \label{sec:commands-tping} The \cmd{tping} command can be used to verify the functionality of a Open MPI universe. It is used to send a ping message between the Open MPI daemons that constitute the Open MPI RTE. It commonly takes two arguments: the set of nodes to ping (expressed in \cmdarg{N} notation) and how many times to ping them. Similar to the Unix \cmd{ping} command, if the number of times to ping is not specified, \cmd{tping} will continue until it is stopped (usually by the user hitting Control-C). The following example pings all nodes in the Open MPI universe three times: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ tping N -c 3 1 byte from 3 remote nodes and 1 local node: 0.002 secs 1 byte from 3 remote nodes and 1 local node: 0.001 secs 1 byte from 3 remote nodes and 1 local node: 0.001 secs 3 messages, 3 bytes (0.003K), 0.005 secs (1.250K/sec) roundtrip min/avg/max: 0.001/0.002/0.002 \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{The \icmd{lamwipe} Command} \label{sec:commands-lamwipe} \changebegin{7.1} The \cmd{lamwipe} command used to be called \idepcmd{wipe}. The name \idepcmd{wipe} has now been deprecated and although it still works in this version of Open MPI, will be removed in future versions. All users are encouraged to start using \cmd{lamwipe} instead. \changeend{7.1} The \cmd{lamwipe} command is used as a ``last resort'' command, and is typically only necessary if \cmd{lamhalt} fails. This usually only occurs in error conditions, such as if a node fails. The \cmd{lamwipe} command takes most of the same parameters as the \cmd{lamboot} command -- it launches a process on each node in the boot schema to kill the Open MPI RTE on that node. Hence, it should be used with the same (or an equivalent) boot schema file as was used with \cmd{lamboot}.