% -*- latex -*- % % Copyright (c) 2004-2005 The Trustees of Indiana University. % All rights reserved. % Copyright (c) 2004-2005 The Trustees of the University of Tennessee. % All rights reserved. % Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, % University of Stuttgart. All rights reserved. % Copyright (c) 2004-2005 The Regents of the University of California. % All rights reserved. % $COPYRIGHT$ % % Additional copyrights may follow % % $HEADER$ % \chapter{Getting Started with Open MPI} \label{sec:getting-started} This chapter provides a summary tutorial describing some of the high points of using Open MPI. It is not intended as a comprehensive guide; the finer details of some situations will not be explained. However, it is a good step-by-step guide for users who are new to MPI and/or Open MPI. Using Open MPI is conceptually simple: \begin{itemize} \item Optionally launch the Open MPI run-time environment, as known as the Open Run-Time Environment (ORTE) \item Repeat as necessary: \begin{itemize} \item Compile MPI program(s) \item Run MPI program(s) \end{itemize} \item If ORTE was started, shut it down. \end{itemize} The tutorial below will describe each of these steps. {\Huge JMS Needs massive overhauling} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{One-Time Setup} This section describes actions that usually only need to be performed once per user in order to setup LAM to function properly. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Setting the Path} \label{sec:getting-started-path} One of the main requirements for Open MPI to function properly is for the LAM executables to be in your path. This step may vary from site to site; for example, the LAM executables may already be in your path -- consult your local administrator to see if this is the case. {\bf NOTE:} If the LAM executables are already in your path, you can skip this step and proceed to Section~\ref{sec:getting-started-ssi}. In many cases, if your system does not already provide the LAM executables in your path, you can add them by editing your ``dot'' files that are executed automatically by the shell upon login (both interactive and non-interactive logins). Each shell has a different file to edit and corresponding syntax, so you'll need to know which shell you are using. Tables~\ref{tbl:getting-started-shells-interactive} and~\ref{tbl:getting-started-shells-noninteractive} list several common shells and the associated files that are typically used. Consult the documentation for your shell for more information. \begin{table}[htbp] \centering \begin{tabular}{|p{1in}|p{4in}|} \hline \multicolumn{1}{|c|}{Shell name} & \multicolumn{1}{|c|}{Interactive login startup file} \\ % \hline \cmd{sh} (or Bash named ``\cmd{sh}'') & \ifile{.profile} \\ % \hline \cmd{csh} & \ifile{.cshrc} followed by \ifile{.login} \\ % \hline \cmd{tcsh} & \ifile{.tcshrc} if it exists, \ifile{.cshrc} if it does not, followed by \ifile{.login} \\ % \hline \cmd{bash} & \ifile{.bash\_\-profile} if it exists, or \ifile{.bash\_\-login} if it exists, or \ifile{.profile} if it exists (in that order). Note that some Linux distributions automatically come with \ifile{.bash\_\-profile} scripts for users that automatically execute \ifile{.bashrc} as well. Consult the \cmd{bash} manual page for more information. \\ \hline \end{tabular} \caption[List of common shells and the corresponding environment setup files for interactive shells.]{List of common shells and the corresponding environmental setup files commonly used with each for interactive startups (e.g., normal login). All files listed are assumed to be in the \file{\$HOME} directory.} \label{tbl:getting-started-shells-interactive} \end{table} \begin{table}[htbp] \centering \begin{tabular}{|p{1in}|p{4in}|} \hline \multicolumn{1}{|c|}{Shell name} & \multicolumn{1}{|c|}{Non-interactive login startup file} \\ % \hline \cmd{sh} (or Bash named ``\cmd{sh}'') & This shell does not execute any file automatically, so LAM will execute the \file{.profile} script before invoking LAM executables on remote nodes \\ % \hline \cmd{csh} & \ifile{.cshrc} \\ % \hline \cmd{tcsh} & \ifile{.tcshrc} if it exists, \ifile{.cshrc} if it does not \\ % \hline \cmd{bash} & \ifile{.bashrc} if it exists \\ \hline \end{tabular} \caption[List of common shells and the corresponding environment setup files for non-interactive shells.]{List of common shells and the corresponding environmental setup files commonly used with each for non-interactive startups (e.g., normal login). All files listed are assumed to be in the \file{\$HOME} directory.} \label{tbl:getting-started-shells-noninteractive} \end{table} You'll also need to know the directory where LAM was installed. For the purposes of this tutorial, we'll assume that LAM is installed in \file{/usr/local/lam}. And to re-emphasize a critical point: these are only guidelines -- the specifics may vary depending on your local setup. Consult your local system or network administrator for more details. Once you have determined all three pieces of information (what shell you are using, what directory LAM was installed to, and what the appropriate ``dot'' file to edit), open the ``dot'' file in a text editor and follow the general directions listed below: \begin{itemize} \index{shell setup!Bash/Bourne shells} \item For the Bash, Bourne, and Bourne-related shells, add the following lines: \lstset{style=lam-bourne} \begin{lstlisting} PATH=/usr/local/lam/bin:$PATH export PATH \end{lstlisting} % Stupid emacs mode: $ \index{shell setup!C shell (and related)} \item For the C shell and related shells (such as \cmd{tcsh}), add the following line: \lstset{style=lam-shell} \begin{lstlisting} set path = (/usr/local/lam/bin $path) \end{lstlisting} % Stupid emacs mode: $ \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Finding the LAM Manual Pages} \index{manual pages} LAM includes manual pages for all supported MPI functions as well as all of the LAM executables. While this step {\em is not necessary} for correct MPI functionality, it can be helpful when looking for MPI or LAM-specific information. Using Tables~\ref{tbl:getting-started-shells-interactive} and~\ref{tbl:getting-started-shells-noninteractive}, find the right ``dot'' file to edit. Assuming again that LAM was installed to \file{/usr/local/lam}, open the appropriate ``dot'' file in a text editor and follow the general directions listed below: \begin{itemize} \index{shell setup!Bash/Bourne shells} \item For the Bash, Bourne, and Bourne-related shells, add the following lines: \lstset{style=lam-bourne} \begin{lstlisting} MANPATH=/usr/local/lam/man:$MANPATH export MANPATH \end{lstlisting} % Stupid emacs mode: $ \index{shell setup!C shell (and related)} \item For the C shell and related shells (such as \cmd{tcsh}), add the following lines: \lstset{style=lam-shell} \begin{lstlisting} if ($?MANPATH == 0) then setenv MANPATH /usr/local/lam/man else setenv MANPATH /usr/local/lam/man:$MANPATH endif \end{lstlisting} % Stupid emacs mode: $ \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{System Services Interface (SSI)} \label{sec:getting-started-ssi} Open MPI is built around a core of System Services Interface (SSI) plugin modules. SSI allows run-time selection of different underlying services within the Open MPI run-time environment, including tunable parameters that can affect the performance of MPI programs. While this tutorial won't go into much detail about SSI, just be aware that you'll see mention of ``SSI'' in the text below. In a few places, the tutorial passes parameters to various SSI modules through either environment variables and/or the \cmdarg{-ssi} command line parameter to several LAM commands. See other sections in this manual for a more complete description of SSI (Chapter~\ref{sec:ssi}, page~\pageref{sec:ssi}), how it works, and what run-time parameters are available (Chapters~\ref{sec:lam-ssi} and~\ref{sec:mpi-ssi}, pages~\pageref{sec:lam-ssi} and~\pageref{sec:mpi-ssi}, respectively). Also, the \manpage{lamssi(7)}, \manpage{lamssi\_\-boot(7)}, \manpage{lamssi\_\-coll(7)}, \manpage{lamssi\_\-cr(7)}, and \manpage{lamssi\_\-rpi(7)} manual pages each provide additional information on LAM's SSI mechanisms. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{What Does Your Open MPI Installation Support?} Open MPI can be installed with a large number of configuration options. It depends on what choices your system/network administrator made when configuring and installing Open MPI. The \icmd{laminfo} command is provided to show the end-user with information about what the installed Open MPI supports. Running ``\cmd{laminfo}'' (with no arguments) prints a list of LAM's capabilities, including all of its SSI modules. Among other things, this shows what language bindings the installed Open MPI supports, what underlying network transports it supports, and what directory LAM was installed to. The \cmdarg{-parsable} option prints out all the same information, but in a conveniently machine-parsable format (suitable for using with scripts). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Booting the LAM Run-Time Environment} \label{sec:getting-started-booting} \index{booting the LAM run-time environment} Before any MPI programs can be executed, the LAM run-time environment must be launched. This is typically called ``booting LAM.'' A successfully boot process creates an instance of the LAM run-time environment commonly referred to as the ``LAM universe.'' LAM's run-time environment can be executed in many different environments. For example, it can be run interactively on a cluster of workstations (even on a single workstation, perhaps to simulate parallel execution for debugging and/or development). Or LAM can be run in production batch scheduled systems. This example will focus on a traditional \cmd{rsh} / \cmd{ssh}-style workstation cluster (i.e., not under batch systems), where \cmd{rsh} or \cmd{ssh} is used to launch executables on remote workstations. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{The Boot Schema File (a.k.a, ``Hostfile'', ``Machinefile'')} \label{sec:getting-started-hostfile} When using \cmd{rsh} or \cmd{ssh} to boot LAM, you will need a text file listing the hosts on which to launch the LAM run-time environment. This file is typically referred to as a ``boot schema'', ``hostfile'', or ``machinefile.'' For example: \lstset{style=lam-shell} \begin{lstlisting} # My boot schema node1.cluster.example.com node2.cluster.example.com node3.cluster.example.com cpu=2 node4.cluster.example.com cpu=2 \end{lstlisting} Four nodes are specified in the above example by listing their IP hostnames. Note also the ``{\tt cpu=2}'' that follows the last two entries. This tells LAM that these machines each have two CPUs available for running MPI programs (e.g., \host{node3} and \host{node4} are two-way SMPs). It is important to note that the number of CPUs specified here has {\em no} correlation to the physicial number of CPUs in the machine. It is simply a convenience mechanism telling LAM how many MPI processes we will typically launch on that node. The ramifications of the {\tt cpu} key will be discussed later. The location of this text file is irrelevant; for the purposes of this example, we'll assume that it is named \file{hostfile} and is located in the current working directory. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{The \icmd{lamboot} Command} \label{sec:getting-started-lamboot} The \cmd{lamboot} command is used to launch the LAM run-time environment. For each machine listed in the boot schema, the following conditions must be met for LAM's run-time environment to be booted correctly: \cmdindex{lamboot}{conditions for success} \begin{itemize} \item The machine must be reachable and operational. \item The user must be able to non-interactively execute arbitrary commands on the machine (e.g., without being prompted for a password). \item The LAM executables must be locatable on that machine, using the user's shell search path. \item The user must be able to write to the LAM session directory (usually somewhere under \file{/tmp}). \item The shell's start-up scripts must not print anything on standard error. \item All machines must be able to resolve the fully-qualified domain name (FQDN) of all the machines being booted (including itself). \end{itemize} Once all of these conditions are met, the \cmd{lamboot} command is used to launch the LAM run-time environment. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamboot -v -ssi boot rsh hostfile LAM 7.0/MPI 2 C++/ROMIO - Indiana University n0<1234> ssi:boot:base:linear: booting n0 (node1.cluster.example.com) n0<1234> ssi:boot:base:linear: booting n1 (node2.cluster.example.com) n0<1234> ssi:boot:base:linear: booting n2 (node3.cluster.example.com) n0<1234> ssi:boot:base:linear: booting n3 (node4.cluster.example.com) n0<1234> ssi:boot:base:linear: finished \end{lstlisting} % Stupid emacs mode: $ The parameters passed to \cmd{lamboot} in the example above are as follows: \begin{itemize} \item \cmdarg{-v}: Make \cmd{lamboot} be slightly verbose. \item \cmdarg{-ssi boot rsh}: Ensure that LAM uses the \cmd{rsh}/\cmd{ssh} boot module to boot the LAM universe. Typically, LAM chooses the right boot module automatically (and therefore this parameter is not typically necessary), but to ensure that this tutorial does exactly what we want it to do, we use this parameter to absolutely ensure that LAM uses \cmd{rsh} or \cmd{ssh} to boot the universe. \item \file{hostfile}: Name of the boot schema file. \end{itemize} Common causes of failure with the \cmd{lamboot} command include (but are not limited to): \cmdindex{lamboot}{common problems and solutions} \begin{itemize} \item User does not have permission to execute on the remote node. This typically involves setting up a \file{\$HOME/.rhosts} file (if using \cmd{rsh}), or properly configured SSH keys (using using \cmd{ssh}). Setting up \file{.rhosts} and/or SSH keys for password-less remote logins are beyond the scope of this tutorial; consult local documentation for \cmd{rsh} and \cmd{ssh}, and/or internet tutorials on setting up SSH keys.\footnote{As of this writing, a Google search for ``ssh keys'' turned up several decent tutorials; including any one of them here would significantly increase the length of this already-tremendously-long manual.} \item The first time a user uses \cmd{ssh} to execute on a remote node, \cmd{ssh} typically prints a warning to the standard error. LAM will interpret this as a failure. If this happens, \cmd{lamboot} will complain that something unexpectedly appeared on \file{stderr}, and abort. % \changebegin{7.1} % One solution is to manually \cmd{ssh} to each node in the boot schema once in order to eliminate the \file{stderr} warning, and then try \cmd{lamboot} again. Another is to use the \ssiparam{boot\_\-rsh\_\-ignore\_\-stderr} SSI parameter. We haven't discussed SSI parameters yet, so it is probably easiest at this point to manually \cmd{ssh} to a small number of nodes to get the warning out of the way. % \changeend{7.1} \end{itemize} If you have having problems with \cmd{lamboot}, try using the \cmdarg{-d} option to \cmd{lamboot}, which will print enormous amounts of debugging output which can be helpful for determining what the problem is. Additionally, check the \file{lamboot(1)} man page as well as the LAM FAQ on the main LAM web site\footnote{\url{http://www.lam-mpi.org/faq/}} under the section ``Booting LAM'' for more information. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{The \icmd{lamnodes} Command} An easy way to see how many nodes and CPUs are in the current LAM universe is with the \cmd{lamnodes} command. For example, with the LAM universe that was created from the boot schema in Section~\ref{sec:getting-started-hostfile}, running the \cmd{lamnodes} command would result in the following output: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamnodes n0 node1.cluster.example.com:1:origin,this_node n1 node2.cluster.example.com:1: n2 node3.cluster.example.com:2: n3 node4.cluster.example.com:2: \end{lstlisting} % Stupid emacs mode: $ The ``{\tt n}'' number on the far left is the LAM node number. For example, ``{\tt n3}'' uniquely refers to \host{node4}. Also note the third column, which indicates how many CPUs are available for running processes on that node. In this example, there are a total of 6 CPUs available for running processes. This information is from the ``{\tt cpu}'' key that was used in the hostfile, and is helpful for running parallel processes (see below). Finally, the ``{\tt origin}'' notation indicates which node \cmd{lamboot} was executed from. ``{\tt this\_\-node}'' obviously indicates which node \cmd{lamnodes} is running on. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Compiling MPI Programs} \label{sec:getting-started-compiling} \index{compiling MPI programs} Note that it is {\em not} necessary to have LAM booted to compile MPI programs. Compiling MPI programs can be a complicated process: \begin{itemize} \item The same compilers should be used to compile/link user MPI programs as were used to compile LAM itself. \item Depending on the specific installation configuration of LAM, a variety of \cmdarg{-I}, \cmdarg{-L}, and \cmdarg{-l} flags (and possibly others) may be necessary to compile and/or link a user MPI program. \end{itemize} Open MPI provides ``wrapper'' compilers to hide all of this complexity. These wrapper compilers simply add the correct compiler/linker flags and then invoke the underlying compiler to actually perform the compilation/link. As such, LAM's wrapper compilers can be used just like ``real'' compilers. The wrapper compilers are named \icmd{mpicc} (for C programs), \icmd{mpiCC} and \icmd{mpic++} (for C++ programs), and \icmd{mpif77} (for Fortran programs). For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc -g -c foo.c shell$ mpicc -g -c bar.c shell$ mpicc -g foo.o bar.o -o my_mpi_program \end{lstlisting} % Stupid emacs mode: $ Note that no additional compiler and linker flags are required for correct MPI compilation or linking. The resulting \cmd{my\_\-mpi\_\-program} is ready to run in the LAM run-time environment. Similarly, the other two wrapper compilers can be used to compile MPI programs for their respective languages: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiCC -O c++_program.cc -o my_c++_mpi_program shell$ mpif77 -O f77_program.f -o my_f77_mpi_program \end{lstlisting} % Stupid emacs mode: $ Note, too, that any other compiler/linker flags can be passed through the wrapper compilers (such as \cmdarg{-g} and \cmdarg{-O}); they will simply be passed to the back-end compiler. Finally, note that giving the \cmdarg{-showme} option to any of the wrapper compilers will show both the name of the back-end compiler that will be invoked, and also all the command line options that would have been passed for a given compile command. For example (line breaks added to fit in the documentation): \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiCC -O c++_program.cc -o my_c++_program -showme g++ -I/usr/local/lam/include -pthread -O c++_program.cc -o \ my_c++_program -L/usr/local/lam/lib -llammpio -llammpi++ -lpmpi \ -llamf77mpi -lmpi -llam -lutil -pthread \end{lstlisting} % Stupid emacs mode: $ \changebegin{7.1} Note that the wrapper compilers only add all the Open MPI-specific flags when a command-line argument that does not begin with a dash (``-'') is present. For example: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc gcc: no input files shell$ mpicc --version gcc (GCC) 3.2.2 (Mandrake Linux 9.1 3.2.2-3mdk) Copyright (C) 2002 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. \end{lstlisting} \changeend{7.1} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Sample MPI Program in C} \index{sample MPI program!C} The following is a simple ``hello world'' C program. \lstset{style=lam-c} \begin{lstlisting} #include #include int main(int argc, char *argv[]) { int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf(``Hello, world! I am %d of %d\n'', rank, size); MPI_Finalize(); return 0; } \end{lstlisting} This program can be saved in a text file and compiled with the \icmd{mpicc} wrapper compiler. \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpicc hello.c -o hello \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Sample MPI Program in C++} \index{sample MPI program!C++} The following is a simple ``hello world'' C++ program. \lstset{style=lam-cxx} \begin{lstlisting} #include #include using namespace std; int main(int argc, char *argv[]) { int rank, size; MPI::Init(argc, argv); rank = MPI::COMM_WORLD.Get_rank(); size = MPI::COMM_WORLD.Get_size(); cout << ``Hello, world! I am '' << rank << `` of '' << size << endl; MPI::Finalize(); return 0; } \end{lstlisting} This program can be saved in a text file and compiled with the \icmd{mpiCC} wrapper compiler (or \cmd{mpic++} if on case-insensitive filesystems, such as Mac OS X's HFS+). \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiCC hello.cc -o hello \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Sample MPI Program in Fortran} \index{sample MPI program!Fortran} The following is a simple ``hello world'' Fortran program. \lstset{style=lam-fortran} \begin{lstlisting} program hello include 'mpif.h' integer rank, size, ierr call MPI_INIT(ierr) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr) call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr) print *, "Hello, world! I am ", rank, " of ", size call MPI_FINALIZE(ierr) stop end \end{lstlisting} This program can be saved in a text file and compiled with the \icmd{mpif77} wrapper compiler. \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpif77 hello.f -o hello \end{lstlisting} % Stupid emacs mode: $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Running MPI Programs} \index{running MPI programs} Once you have successfully established a LAM universe and compiled an MPI program, you can run MPI programs in parallel. In this section, we will show how to run a Single Program, Multiple Data (SPMD) program. Specifically, we will run the \cmd{hello} program (from the previous section) in parallel. The \cmd{mpirun} and \cmd{mpiexec} commands are used for launching parallel MPI programs, and the \cmd{mpitask} commands can be used to provide crude debugging support. The \cmd{lamclean} command can be used to completely clean up a failed MPI program (e.g., if an error occurs). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{The \icmd{mpirun} Command} The \cmd{mpirun} command has many different options that can be used to control the execution of a program in parallel. We'll explain only a few of them here. The simplest way to launch the \cmd{hello} program across all CPUs listed in the boot schema is: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpirun C hello \end{lstlisting} % stupid emacs mode: $ The \cmdarg{C} option means ``launch one copy of \cmd{hello} on every CPU that was listed in the boot schema.'' The \cmdarg{C} notation is therefore convenient shorthand notation for launching a set of processes across a group of SMPs. Another method for running in parallel is: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpirun N hello \end{lstlisting} % stupid emacs mode: $ The \cmdarg{N} option has a different meaning than \cmdarg{C} -- it means ``launch one copy of \cmd{hello} on every node in the LAM universe.'' Hence, \cmdarg{N} disregards the CPU count. This can be useful for multi-threaded MPI programs. Finally, to run an absolute number of processes (regardless of how many CPUs or nodes are in the LAM universe): \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpirun -np 4 hello \end{lstlisting} % stupid emacs mode: $ This runs 4 copies of \cmd{hello}. LAM will ``schedule'' how many copies of \cmd{hello} will be run in a round-robin fashion on each node by how many CPUs were listed in the boot schema file.\footnote{Note that the use of the word ``schedule'' does not imply that LAM has ties with the operating system for scheduling purposes (it doesn't). LAM ``scheduled'' on a per-node basis; so selecting a process to run means that it has been assigned and launched on that node. The operating system is solely responsible for all process and kernel scheduling.} For example, on the LAM universe that we have previously shown in this tutorial, the following would be launched: \begin{itemize} \item 1 \cmd{hello} would be launched on {\tt n0} (named \host{node1}) \item 1 \cmd{hello} would be launched on {\tt n1} (named \host{node2}) \item 2 \cmd{hello}s would be launched on {\tt n2} (named \host{node3}) \end{itemize} Note that any number can be used -- if a number is used that is greater than how many CPUs are in the LAM universe, LAM will ``wrap around'' and start scheduling starting with the first node again. For example, using \cmdarg{-np 10} would result in the following schedule: \begin{itemize} \item 2 \cmd{hello}s on {\tt n0} (1 from the first pass, and then a second from the ``wrap around'') \item 2 \cmd{hello}s on {\tt n1} (1 from the first pass, and then a second from the ``wrap around'') \item 4 \cmd{hello}s on {\tt n2} (2 from the first pass, and then 2 more from the ``wrap around'') \item 2 \cmd{hello}s on {\tt n3} \end{itemize} The \file{mpirun(1)} man page contains much more information and \cmd{mpirun} and the options available. For example, \cmd{mpirun} also supports Multiple Program, Multiple Data (MPMD) programs, although it is not discussed here. Also see Section~\ref{sec:commands-mpirun} (page~\pageref{sec:commands-mpirun}) in this document. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{The \icmd{mpiexec} Command} The MPI-2 standard recommends the use of \cmd{mpiexec} for portable MPI process startup. In Open MPI, \cmd{mpiexec} is functionaly similar to \cmd{mpirun}. Some options that are available to \cmd{mpirun} are not available to \cmd{mpiexec}, and vice-versa. The end result is typically the same, however -- both will launch parallel MPI programs; which you should use is likely simply a personal choice. That being said, \cmd{mpiexec} offers more convenient access in three cases: \begin{itemize} \item Running MPMD programs \item Running heterogeneous programs \item Running ``one-shot'' MPI programs (i.e., boot LAM, run the program, then halt LAM) \end{itemize} The general syntax for \cmd{mpiexec} is: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec : : ... \end{lstlisting} % stupid emacs mode: $ %%%%% \subsubsection{Running MPMD Programs} For example, to run a manager/worker parallel program, where two different executables need to be launched (i.e., \cmd{manager} and \cmd{worker}, the following can be used: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -n 1 manager : worker \end{lstlisting} % stupid emacs mode: $ This runs one copy of \cmd{manager} and one copy of \cmd{worker} for every CPU in the LAM universe. %%%%% \subsubsection{Running Heterogeneous Programs} Since LAM is a heterogeneous MPI implementation, it supports running heterogeneous MPI programs. For example, this allows running a parallel job that spans a Sun SPARC machine and an IA-32 Linux machine (even though they are opposite endian machines). Although this can be somewhat complicated to setup (remember that you will first need to \cmd{lamboot} successfully, which essentially means that LAM must be correctly installed on both architectures), the \cmd{mpiexec} command can be helpful in actually running the resulting MPI job. Note that you will need to have two MPI executables -- one compiled for Solaris (e.g., \cmd{hello.solaris}) and one compiled for Linux (e.g., \cmd{hello.linux}). Assuming that these executables both reside in the same directory, and that directory is available on both nodes (or the executables can be found in the \envvar{PATH} on their respective machines), the following command can be used: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -arch solaris hello.solaris : -arch linux hello.linux \end{lstlisting} % stupid emacs mode: $ This runs the \cmd{hello.solaris} command on all nodes in the LAM universe that have the string ``solaris'' anywhere in their architecture string, and \cmd{hello.linux} on all nodes that have ``linux'' in their architecture string. The architecture string of a given LAM installation can be found by running the \cmd{laminfo} command. %%%%% \subsubsection{``One-Shot'' MPI Programs} In some cases, it seems like extra work to boot a LAM universe, run a single MPI job, and then shut down the universe. Batch jobs are good examples of this -- since only one job is going to be run, why does it take three commands? \cmd{mpiexec} provides a convenient way to run ``one-shot'' MPI jobs. \lstset{style=lam-cmdline} \begin{lstlisting} shell$ mpiexec -machinefile hostfile hello \end{lstlisting} % stupid emacs mode: $ This will invoke \cmd{lamboot} with the boot schema named ``\file{hostfile}'', run the MPI program \cmd{hello} on all available CPUs in the resulting universe, and then shut down the universe with the \cmd{lamhalt} command (which we'll discuss in Section~\ref{sec:getting-started-lamhalt}, below). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{The \icmd{mpitask} Command} The \cmd{mpitask} command is analogous to the sequential Unix command \cmd{ps}. It shows the current status of the MPI program(s) being executed in the LAM universe, and displays primitive information about what MPI function each process is currently executing (if any). Note that in normal practice, the \cmd{mpimsg} command only gives a snapshot of what messages are flowing between MPI processes, and therefore is usually only accurate at that single point in time. To really debug message passing traffic, use a tool such as message passing analyzer (e.g., XMPI), or a parallel debugger (e.g., TotalView). \cmd{mpitask} can be run from any node in the LAM universe. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{The \icmd{lamclean} Command} The \cmd{lamclean} command completely removed all running programs from the LAM universe. This can be useful if a parallel job crashes and/or leaves state in the LAM run-time environment (e.g., MPI-2 published names). It is usually run with no parameters: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamclean \end{lstlisting} % stupid emacs mode: $ \cmd{lamclean} is typically only necessary when developing / debugging MPI applications -- i.e., programs that hang, messages that are left around, etc. Correct MPI programs should terminate properly, clean up all their messages, unpublish MPI-2 names, etc. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Shutting Down the LAM Universe} \label{sec:getting-started-lamhalt} When finished with the LAM universe, it should be shut down with the \icmd{lamhalt} command: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamhalt \end{lstlisting} % Stupid emacs mode: $ In most cases, this is sufficient to kill all running MPI processes and shut down the LAM universe. However, in some rare conditions, \cmd{lamhalt} may fail. For example, if any of the nodes in the LAM universe crashed before running \cmd{lamhalt}, \cmd{lamhalt} will likely timeout and potentially not kill the entire LAM universe. In this case, you will need to use the \icmd{lamwipe} command to guarantee that the LAM universe has shut down properly: \lstset{style=lam-cmdline} \begin{lstlisting} shell$ lamwipe -v hostfile \end{lstlisting} % Stupid emacs mode: $ \noindent where \file{hostfile} is the same boot schema that was used to boot LAM (i.e., all the same nodes are listed). \cmd{lamwipe} will forcibly kill all Open MPI processes and terminate the LAM universe. This is a slower process than \cmd{lamhalt}, and is typically not necessary.