949 строки
34 KiB
TeX
949 строки
34 KiB
TeX
% -*- latex -*-
|
|
%
|
|
% Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
|
% University Research and Technology
|
|
% Corporation. All rights reserved.
|
|
% Copyright (c) 2004-2005 The University of Tennessee and The University
|
|
% of Tennessee Research Foundation. All rights
|
|
% reserved.
|
|
% Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
% University of Stuttgart. All rights reserved.
|
|
% Copyright (c) 2004-2005 The Regents of the University of California.
|
|
% All rights reserved.
|
|
% $COPYRIGHT$
|
|
%
|
|
% Additional copyrights may follow
|
|
%
|
|
% $HEADER$
|
|
%
|
|
|
|
\chapter{Getting Started with Open MPI}
|
|
\label{sec:getting-started}
|
|
|
|
This chapter provides a summary tutorial describing some of the high
|
|
points of using Open MPI. It is not intended as a comprehensive
|
|
guide; the finer details of some situations will not be explained.
|
|
However, it is a good step-by-step guide for users who are new to MPI
|
|
and/or Open MPI.
|
|
|
|
Using Open MPI is conceptually simple:
|
|
|
|
\begin{itemize}
|
|
\item Optionally launch the Open MPI run-time environment, as known as
|
|
the Open Run-Time Environment (ORTE)
|
|
\item Repeat as necessary:
|
|
\begin{itemize}
|
|
\item Compile MPI program(s)
|
|
\item Run MPI program(s)
|
|
\end{itemize}
|
|
\item If ORTE was started, shut it down.
|
|
\end{itemize}
|
|
|
|
The tutorial below will describe each of these steps.
|
|
|
|
{\Huge JMS Needs massive overhauling}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{One-Time Setup}
|
|
|
|
This section describes actions that usually only need to be performed
|
|
once per user in order to setup LAM to function properly.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{Setting the Path}
|
|
\label{sec:getting-started-path}
|
|
|
|
One of the main requirements for Open MPI to function properly is for
|
|
the LAM executables to be in your path. This step may vary from site
|
|
to site; for example, the LAM executables may already be in your path
|
|
-- consult your local administrator to see if this is the case.
|
|
|
|
{\bf NOTE:} If the LAM executables are already in your path, you can
|
|
skip this step and proceed to
|
|
Section~\ref{sec:getting-started-ssi}.
|
|
|
|
In many cases, if your system does not already provide the LAM
|
|
executables in your path, you can add them by editing your ``dot''
|
|
files that are executed automatically by the shell upon login (both
|
|
interactive and non-interactive logins). Each shell has a different
|
|
file to edit and corresponding syntax, so you'll need to know which
|
|
shell you are using.
|
|
Tables~\ref{tbl:getting-started-shells-interactive}
|
|
and~\ref{tbl:getting-started-shells-noninteractive} list several
|
|
common shells and the associated files that are typically used.
|
|
Consult the documentation for your shell for more information.
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\begin{tabular}{|p{1in}|p{4in}|}
|
|
\hline
|
|
\multicolumn{1}{|c|}{Shell name} &
|
|
\multicolumn{1}{|c|}{Interactive login startup file} \\
|
|
%
|
|
\hline
|
|
\cmd{sh} (or Bash named ``\cmd{sh}'') & \ifile{.profile} \\
|
|
%
|
|
\hline
|
|
\cmd{csh} & \ifile{.cshrc} followed by \ifile{.login} \\
|
|
%
|
|
\hline
|
|
\cmd{tcsh} & \ifile{.tcshrc} if it exists, \ifile{.cshrc} if it
|
|
does not, followed by \ifile{.login} \\
|
|
%
|
|
\hline
|
|
\cmd{bash} & \ifile{.bash\_\-profile} if it exists, or
|
|
\ifile{.bash\_\-login} if it exists, or \ifile{.profile} if it
|
|
exists (in that order). Note that some Linux distributions
|
|
automatically come with \ifile{.bash\_\-profile} scripts for users
|
|
that automatically execute \ifile{.bashrc} as well. Consult the
|
|
\cmd{bash} manual page for more information. \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption[List of common shells and the corresponding environment
|
|
setup files for interactive shells.]{List of common shells and
|
|
the corresponding environmental setup files commonly used with
|
|
each for interactive startups (e.g., normal login). All files
|
|
listed are assumed to be in the \file{\$HOME} directory.}
|
|
\label{tbl:getting-started-shells-interactive}
|
|
\end{table}
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\begin{tabular}{|p{1in}|p{4in}|}
|
|
\hline
|
|
\multicolumn{1}{|c|}{Shell name} &
|
|
\multicolumn{1}{|c|}{Non-interactive login startup file} \\
|
|
%
|
|
\hline
|
|
\cmd{sh} (or Bash named ``\cmd{sh}'') & This shell does not
|
|
execute any file automatically, so LAM will execute the
|
|
\file{.profile} script before invoking LAM executables on remote
|
|
nodes \\
|
|
%
|
|
\hline
|
|
\cmd{csh} & \ifile{.cshrc} \\
|
|
%
|
|
\hline
|
|
\cmd{tcsh} & \ifile{.tcshrc} if it exists, \ifile{.cshrc} if it
|
|
does not \\
|
|
%
|
|
\hline
|
|
\cmd{bash} & \ifile{.bashrc} if it exists \\
|
|
\hline
|
|
\end{tabular}
|
|
\caption[List of common shells and the corresponding environment
|
|
setup files for non-interactive shells.]{List of common shells and
|
|
the corresponding environmental setup files commonly used with
|
|
each for non-interactive startups (e.g., normal login). All files
|
|
listed are assumed to be in the \file{\$HOME} directory.}
|
|
\label{tbl:getting-started-shells-noninteractive}
|
|
\end{table}
|
|
|
|
You'll also need to know the directory where LAM was installed. For
|
|
the purposes of this tutorial, we'll assume that LAM is installed in
|
|
\file{/usr/local/lam}. And to re-emphasize a critical point: these
|
|
are only guidelines -- the specifics may vary depending on your local
|
|
setup. Consult your local system or network administrator for more
|
|
details.
|
|
|
|
Once you have determined all three pieces of information (what shell
|
|
you are using, what directory LAM was installed to, and what the
|
|
appropriate ``dot'' file to edit), open the ``dot'' file in a text
|
|
editor and follow the general directions listed below:
|
|
|
|
\begin{itemize}
|
|
\index{shell setup!Bash/Bourne shells}
|
|
\item For the Bash, Bourne, and Bourne-related shells, add the
|
|
following lines:
|
|
|
|
\lstset{style=lam-bourne}
|
|
\begin{lstlisting}
|
|
PATH=/usr/local/lam/bin:$PATH
|
|
export PATH
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
\index{shell setup!C shell (and related)}
|
|
\item For the C shell and related shells (such as \cmd{tcsh}), add the
|
|
following line:
|
|
|
|
\lstset{style=lam-shell}
|
|
\begin{lstlisting}
|
|
set path = (/usr/local/lam/bin $path)
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
\end{itemize}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{Finding the LAM Manual Pages}
|
|
\index{manual pages}
|
|
|
|
LAM includes manual pages for all supported MPI functions as well as
|
|
all of the LAM executables. While this step {\em is not necessary}
|
|
for correct MPI functionality, it can be helpful when looking for MPI
|
|
or LAM-specific information.
|
|
|
|
Using Tables~\ref{tbl:getting-started-shells-interactive}
|
|
and~\ref{tbl:getting-started-shells-noninteractive}, find the right
|
|
``dot'' file to edit. Assuming again that LAM was installed to
|
|
\file{/usr/local/lam}, open the appropriate ``dot'' file in a text
|
|
editor and follow the general directions listed below:
|
|
|
|
\begin{itemize}
|
|
\index{shell setup!Bash/Bourne shells}
|
|
\item For the Bash, Bourne, and Bourne-related shells, add the
|
|
following lines:
|
|
|
|
\lstset{style=lam-bourne}
|
|
\begin{lstlisting}
|
|
MANPATH=/usr/local/lam/man:$MANPATH
|
|
export MANPATH
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
\index{shell setup!C shell (and related)}
|
|
\item For the C shell and related shells (such as \cmd{tcsh}), add the
|
|
following lines:
|
|
|
|
\lstset{style=lam-shell}
|
|
\begin{lstlisting}
|
|
if ($?MANPATH == 0) then
|
|
setenv MANPATH /usr/local/lam/man
|
|
else
|
|
setenv MANPATH /usr/local/lam/man:$MANPATH
|
|
endif
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
\end{itemize}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{System Services Interface (SSI)}
|
|
\label{sec:getting-started-ssi}
|
|
|
|
Open MPI is built around a core of System Services Interface (SSI)
|
|
plugin modules. SSI allows run-time selection of different underlying
|
|
services within the Open MPI run-time environment, including tunable
|
|
parameters that can affect the performance of MPI programs.
|
|
|
|
While this tutorial won't go into much detail about SSI, just be aware
|
|
that you'll see mention of ``SSI'' in the text below. In a few
|
|
places, the tutorial passes parameters to various SSI modules through
|
|
either environment variables and/or the \cmdarg{-ssi} command line
|
|
parameter to several LAM commands.
|
|
|
|
See other sections in this manual for a more complete description of
|
|
SSI (Chapter~\ref{sec:ssi}, page~\pageref{sec:ssi}), how it works, and
|
|
what run-time parameters are available (Chapters~\ref{sec:lam-ssi}
|
|
and~\ref{sec:mpi-ssi}, pages~\pageref{sec:lam-ssi}
|
|
and~\pageref{sec:mpi-ssi}, respectively). Also, the
|
|
\manpage{lamssi(7)}, \manpage{lamssi\_\-boot(7)},
|
|
\manpage{lamssi\_\-coll(7)}, \manpage{lamssi\_\-cr(7)}, and
|
|
\manpage{lamssi\_\-rpi(7)} manual pages each provide additional
|
|
information on LAM's SSI mechanisms.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{What Does Your Open MPI Installation Support?}
|
|
|
|
Open MPI can be installed with a large number of configuration options.
|
|
It depends on what choices your system/network administrator made when
|
|
configuring and installing Open MPI. The \icmd{laminfo} command is
|
|
provided to show the end-user with information about what the
|
|
installed Open MPI supports. Running ``\cmd{laminfo}'' (with no
|
|
arguments) prints a list of LAM's capabilities, including all of its
|
|
SSI modules.
|
|
|
|
Among other things, this shows what language bindings the installed
|
|
Open MPI supports, what underlying network transports it supports, and
|
|
what directory LAM was installed to. The \cmdarg{-parsable} option
|
|
prints out all the same information, but in a conveniently
|
|
machine-parsable format (suitable for using with scripts).
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Booting the LAM Run-Time Environment}
|
|
\label{sec:getting-started-booting}
|
|
\index{booting the LAM run-time environment}
|
|
|
|
Before any MPI programs can be executed, the LAM run-time environment
|
|
must be launched. This is typically called ``booting LAM.'' A
|
|
successfully boot process creates an instance of the LAM run-time
|
|
environment commonly referred to as the ``LAM universe.''
|
|
|
|
LAM's run-time environment can be executed in many different
|
|
environments. For example, it can be run interactively on a cluster
|
|
of workstations (even on a single workstation, perhaps to simulate
|
|
parallel execution for debugging and/or development). Or LAM can be
|
|
run in production batch scheduled systems.
|
|
|
|
This example will focus on a traditional \cmd{rsh} / \cmd{ssh}-style
|
|
workstation cluster (i.e., not under batch systems), where \cmd{rsh}
|
|
or \cmd{ssh} is used to launch executables on remote workstations.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{The Boot Schema File (a.k.a, ``Hostfile'', ``Machinefile'')}
|
|
\label{sec:getting-started-hostfile}
|
|
|
|
When using \cmd{rsh} or \cmd{ssh} to boot LAM, you will need a text
|
|
file listing the hosts on which to launch the LAM run-time
|
|
environment. This file is typically referred to as a ``boot schema'',
|
|
``hostfile'', or ``machinefile.'' For example:
|
|
|
|
\lstset{style=lam-shell}
|
|
\begin{lstlisting}
|
|
# My boot schema
|
|
node1.cluster.example.com
|
|
node2.cluster.example.com
|
|
node3.cluster.example.com cpu=2
|
|
node4.cluster.example.com cpu=2
|
|
\end{lstlisting}
|
|
|
|
Four nodes are specified in the above example by listing their IP
|
|
hostnames. Note also the ``{\tt cpu=2}'' that follows the last two
|
|
entries. This tells LAM that these machines each have two CPUs
|
|
available for running MPI programs (e.g., \host{node3} and
|
|
\host{node4} are two-way SMPs). It is important to note that the
|
|
number of CPUs specified here has {\em no} correlation to the
|
|
physicial number of CPUs in the machine. It is simply a convenience
|
|
mechanism telling LAM how many MPI processes we will typically launch
|
|
on that node. The ramifications of the {\tt cpu} key will be discussed
|
|
later.
|
|
|
|
The location of this text file is irrelevant; for the purposes of this
|
|
example, we'll assume that it is named \file{hostfile} and is located
|
|
in the current working directory.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{The \icmd{lamboot} Command}
|
|
\label{sec:getting-started-lamboot}
|
|
|
|
The \cmd{lamboot} command is used to launch the LAM run-time
|
|
environment. For each machine listed in the boot schema, the
|
|
following conditions must be met for LAM's run-time environment to be
|
|
booted correctly:
|
|
|
|
\cmdindex{lamboot}{conditions for success}
|
|
\begin{itemize}
|
|
\item The machine must be reachable and operational.
|
|
|
|
\item The user must be able to non-interactively execute arbitrary
|
|
commands on the machine (e.g., without being prompted for a
|
|
password).
|
|
|
|
\item The LAM executables must be locatable on that machine, using the
|
|
user's shell search path.
|
|
|
|
\item The user must be able to write to the LAM session directory
|
|
(usually somewhere under \file{/tmp}).
|
|
|
|
\item The shell's start-up scripts must not print anything on standard
|
|
error.
|
|
|
|
\item All machines must be able to resolve the fully-qualified domain
|
|
name (FQDN) of all the machines being booted (including itself).
|
|
\end{itemize}
|
|
|
|
Once all of these conditions are met, the \cmd{lamboot} command is
|
|
used to launch the LAM run-time environment. For example:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ lamboot -v -ssi boot rsh hostfile
|
|
|
|
LAM 7.0/MPI 2 C++/ROMIO - Indiana University
|
|
|
|
n0<1234> ssi:boot:base:linear: booting n0 (node1.cluster.example.com)
|
|
n0<1234> ssi:boot:base:linear: booting n1 (node2.cluster.example.com)
|
|
n0<1234> ssi:boot:base:linear: booting n2 (node3.cluster.example.com)
|
|
n0<1234> ssi:boot:base:linear: booting n3 (node4.cluster.example.com)
|
|
n0<1234> ssi:boot:base:linear: finished
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
The parameters passed to \cmd{lamboot} in the example above are as
|
|
follows:
|
|
|
|
\begin{itemize}
|
|
\item \cmdarg{-v}: Make \cmd{lamboot} be slightly verbose.
|
|
|
|
\item \cmdarg{-ssi boot rsh}: Ensure that LAM uses the
|
|
\cmd{rsh}/\cmd{ssh} boot module to boot the LAM universe.
|
|
Typically, LAM chooses the right boot module automatically (and
|
|
therefore this parameter is not typically necessary), but to ensure
|
|
that this tutorial does exactly what we want it to do, we use this
|
|
parameter to absolutely ensure that LAM uses \cmd{rsh} or \cmd{ssh}
|
|
to boot the universe.
|
|
|
|
\item \file{hostfile}: Name of the boot schema file.
|
|
\end{itemize}
|
|
|
|
Common causes of failure with the \cmd{lamboot} command include (but
|
|
are not limited to):
|
|
|
|
\cmdindex{lamboot}{common problems and solutions}
|
|
\begin{itemize}
|
|
\item User does not have permission to execute on the remote node.
|
|
This typically involves setting up a \file{\$HOME/.rhosts} file (if
|
|
using \cmd{rsh}), or properly configured SSH keys (using using
|
|
\cmd{ssh}).
|
|
|
|
Setting up \file{.rhosts} and/or SSH keys for password-less remote
|
|
logins are beyond the scope of this tutorial; consult local
|
|
documentation for \cmd{rsh} and \cmd{ssh}, and/or internet tutorials
|
|
on setting up SSH keys.\footnote{As of this writing, a Google search
|
|
for ``ssh keys'' turned up several decent tutorials; including any
|
|
one of them here would significantly increase the length of this
|
|
already-tremendously-long manual.}
|
|
|
|
\item The first time a user uses \cmd{ssh} to execute on a remote
|
|
node, \cmd{ssh} typically prints a warning to the standard error.
|
|
LAM will interpret this as a failure. If this happens,
|
|
\cmd{lamboot} will complain that something unexpectedly appeared on
|
|
\file{stderr}, and abort.
|
|
%
|
|
\changebegin{7.1}
|
|
%
|
|
One solution is to manually \cmd{ssh} to each node in the boot
|
|
schema once in order to eliminate the \file{stderr} warning, and
|
|
then try \cmd{lamboot} again. Another is to use the
|
|
\ssiparam{boot\_\-rsh\_\-ignore\_\-stderr} SSI parameter. We
|
|
haven't discussed SSI parameters yet, so it is probably easiest at
|
|
this point to manually \cmd{ssh} to a small number of nodes to get
|
|
the warning out of the way.
|
|
%
|
|
\changeend{7.1}
|
|
\end{itemize}
|
|
|
|
If you have having problems with \cmd{lamboot}, try using the
|
|
\cmdarg{-d} option to \cmd{lamboot}, which will print enormous amounts
|
|
of debugging output which can be helpful for determining what the
|
|
problem is. Additionally, check the \file{lamboot(1)} man page as
|
|
well as the LAM FAQ on the main LAM web
|
|
site\footnote{\url{http://www.lam-mpi.org/faq/}} under the section
|
|
``Booting LAM'' for more information.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{The \icmd{lamnodes} Command}
|
|
|
|
An easy way to see how many nodes and CPUs are in the current LAM
|
|
universe is with the \cmd{lamnodes} command. For example, with the
|
|
LAM universe that was created from the boot schema in
|
|
Section~\ref{sec:getting-started-hostfile}, running the \cmd{lamnodes}
|
|
command would result in the following output:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ lamnodes
|
|
n0 node1.cluster.example.com:1:origin,this_node
|
|
n1 node2.cluster.example.com:1:
|
|
n2 node3.cluster.example.com:2:
|
|
n3 node4.cluster.example.com:2:
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
The ``{\tt n}'' number on the far left is the LAM node number. For
|
|
example, ``{\tt n3}'' uniquely refers to \host{node4}. Also note the
|
|
third column, which indicates how many CPUs are available for running
|
|
processes on that node. In this example, there are a total of 6 CPUs
|
|
available for running processes. This information is from the ``{\tt
|
|
cpu}'' key that was used in the hostfile, and is helpful for running
|
|
parallel processes (see below).
|
|
|
|
Finally, the ``{\tt origin}'' notation indicates which node
|
|
\cmd{lamboot} was executed from. ``{\tt this\_\-node}'' obviously
|
|
indicates which node \cmd{lamnodes} is running on.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Compiling MPI Programs}
|
|
\label{sec:getting-started-compiling}
|
|
\index{compiling MPI programs}
|
|
|
|
Note that it is {\em not} necessary to have LAM booted to compile MPI
|
|
programs.
|
|
|
|
Compiling MPI programs can be a complicated process:
|
|
|
|
\begin{itemize}
|
|
\item The same compilers should be used to compile/link user MPI
|
|
programs as were used to compile LAM itself.
|
|
|
|
\item Depending on the specific installation configuration of LAM, a
|
|
variety of \cmdarg{-I}, \cmdarg{-L}, and \cmdarg{-l} flags (and
|
|
possibly others) may be necessary to compile and/or link a user MPI
|
|
program.
|
|
\end{itemize}
|
|
|
|
Open MPI provides ``wrapper'' compilers to hide all of this complexity.
|
|
These wrapper compilers simply add the correct compiler/linker flags
|
|
and then invoke the underlying compiler to actually perform the
|
|
compilation/link. As such, LAM's wrapper compilers can be used just
|
|
like ``real'' compilers.
|
|
|
|
The wrapper compilers are named \icmd{mpicc} (for C programs),
|
|
\icmd{mpiCC} and \icmd{mpic++} (for C++ programs), and \icmd{mpif77}
|
|
(for Fortran programs). For example:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpicc -g -c foo.c
|
|
shell$ mpicc -g -c bar.c
|
|
shell$ mpicc -g foo.o bar.o -o my_mpi_program
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
Note that no additional compiler and linker flags are required for
|
|
correct MPI compilation or linking. The resulting
|
|
\cmd{my\_\-mpi\_\-program} is ready to run in the LAM run-time
|
|
environment. Similarly, the other two wrapper compilers can be used
|
|
to compile MPI programs for their respective languages:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpiCC -O c++_program.cc -o my_c++_mpi_program
|
|
shell$ mpif77 -O f77_program.f -o my_f77_mpi_program
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
Note, too, that any other compiler/linker flags can be passed through
|
|
the wrapper compilers (such as \cmdarg{-g} and \cmdarg{-O}); they will
|
|
simply be passed to the back-end compiler.
|
|
|
|
Finally, note that giving the \cmdarg{-showme} option to any of the
|
|
wrapper compilers will show both the name of the back-end compiler
|
|
that will be invoked, and also all the command line options that would
|
|
have been passed for a given compile command. For example (line
|
|
breaks added to fit in the documentation):
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpiCC -O c++_program.cc -o my_c++_program -showme
|
|
g++ -I/usr/local/lam/include -pthread -O c++_program.cc -o \
|
|
my_c++_program -L/usr/local/lam/lib -llammpio -llammpi++ -lpmpi \
|
|
-llamf77mpi -lmpi -llam -lutil -pthread
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
\changebegin{7.1}
|
|
|
|
Note that the wrapper compilers only add all the Open MPI-specific
|
|
flags when a command-line argument that does not begin with a dash
|
|
(``-'') is present. For example:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpicc
|
|
gcc: no input files
|
|
shell$ mpicc --version
|
|
gcc (GCC) 3.2.2 (Mandrake Linux 9.1 3.2.2-3mdk)
|
|
Copyright (C) 2002 Free Software Foundation, Inc.
|
|
This is free software; see the source for copying conditions. There is NO
|
|
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
|
|
\end{lstlisting}
|
|
|
|
\changeend{7.1}
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{Sample MPI Program in C}
|
|
\index{sample MPI program!C}
|
|
|
|
The following is a simple ``hello world'' C program.
|
|
|
|
\lstset{style=lam-c}
|
|
|
|
\begin{lstlisting}
|
|
#include <stdio.h>
|
|
#include <mpi.h>
|
|
|
|
int main(int argc, char *argv[]) {
|
|
int rank, size;
|
|
|
|
MPI_Init(&argc, &argv);
|
|
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
|
|
MPI_Comm_size(MPI_COMM_WORLD, &size);
|
|
|
|
printf(``Hello, world! I am %d of %d\n'', rank, size);
|
|
|
|
MPI_Finalize();
|
|
return 0;
|
|
}
|
|
\end{lstlisting}
|
|
|
|
This program can be saved in a text file and compiled with the
|
|
\icmd{mpicc} wrapper compiler.
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpicc hello.c -o hello
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{Sample MPI Program in C++}
|
|
\index{sample MPI program!C++}
|
|
|
|
The following is a simple ``hello world'' C++ program.
|
|
|
|
\lstset{style=lam-cxx}
|
|
|
|
\begin{lstlisting}
|
|
#include <iostream>
|
|
#include <mpi.h>
|
|
|
|
using namespace std;
|
|
|
|
int main(int argc, char *argv[]) {
|
|
int rank, size;
|
|
|
|
MPI::Init(argc, argv);
|
|
rank = MPI::COMM_WORLD.Get_rank();
|
|
size = MPI::COMM_WORLD.Get_size();
|
|
|
|
cout << ``Hello, world! I am '' << rank << `` of '' << size << endl;
|
|
|
|
MPI::Finalize();
|
|
return 0;
|
|
}
|
|
\end{lstlisting}
|
|
|
|
This program can be saved in a text file and compiled with the
|
|
\icmd{mpiCC} wrapper compiler (or \cmd{mpic++} if on case-insensitive
|
|
filesystems, such as Mac OS X's HFS+).
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpiCC hello.cc -o hello
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{Sample MPI Program in Fortran}
|
|
\index{sample MPI program!Fortran}
|
|
|
|
The following is a simple ``hello world'' Fortran program.
|
|
|
|
\lstset{style=lam-fortran}
|
|
|
|
\begin{lstlisting}
|
|
program hello
|
|
include 'mpif.h'
|
|
integer rank, size, ierr
|
|
|
|
call MPI_INIT(ierr)
|
|
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
|
|
call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr)
|
|
|
|
print *, "Hello, world! I am ", rank, " of ", size
|
|
|
|
call MPI_FINALIZE(ierr)
|
|
stop
|
|
end
|
|
\end{lstlisting}
|
|
|
|
This program can be saved in a text file and compiled with the
|
|
\icmd{mpif77} wrapper compiler.
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpif77 hello.f -o hello
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Running MPI Programs}
|
|
\index{running MPI programs}
|
|
|
|
Once you have successfully established a LAM universe and compiled an
|
|
MPI program, you can run MPI programs in parallel.
|
|
|
|
In this section, we will show how to run a Single Program, Multiple
|
|
Data (SPMD) program. Specifically, we will run the \cmd{hello}
|
|
program (from the previous section) in parallel. The \cmd{mpirun} and
|
|
\cmd{mpiexec} commands are used for launching parallel MPI programs,
|
|
and the \cmd{mpitask} commands can be used to provide crude debugging
|
|
support. The \cmd{lamclean} command can be used to completely clean
|
|
up a failed MPI program (e.g., if an error occurs).
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{The \icmd{mpirun} Command}
|
|
|
|
The \cmd{mpirun} command has many different options that can be used
|
|
to control the execution of a program in parallel. We'll explain only
|
|
a few of them here.
|
|
|
|
The simplest way to launch the \cmd{hello} program across all CPUs
|
|
listed in the boot schema is:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpirun C hello
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
The \cmdarg{C} option means ``launch one copy of \cmd{hello} on
|
|
every CPU that was listed in the boot schema.'' The \cmdarg{C}
|
|
notation is therefore convenient shorthand notation for launching a
|
|
set of processes across a group of SMPs.
|
|
|
|
Another method for running in parallel is:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpirun N hello
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
The \cmdarg{N} option has a different meaning than \cmdarg{C} -- it
|
|
means ``launch one copy of \cmd{hello} on every node in the LAM
|
|
universe.'' Hence, \cmdarg{N} disregards the CPU count. This can be
|
|
useful for multi-threaded MPI programs.
|
|
|
|
Finally, to run an absolute number of processes (regardless of how
|
|
many CPUs or nodes are in the LAM universe):
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpirun -np 4 hello
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
This runs 4 copies of \cmd{hello}. LAM will ``schedule'' how many
|
|
copies of \cmd{hello} will be run in a round-robin fashion on each
|
|
node by how many CPUs were listed in the boot schema
|
|
file.\footnote{Note that the use of the word ``schedule'' does not
|
|
imply that LAM has ties with the operating system for scheduling
|
|
purposes (it doesn't). LAM ``scheduled'' on a per-node basis; so
|
|
selecting a process to run means that it has been assigned and
|
|
launched on that node. The operating system is solely responsible
|
|
for all process and kernel scheduling.} For example, on the LAM
|
|
universe that we have previously shown in this tutorial, the following
|
|
would be launched:
|
|
|
|
\begin{itemize}
|
|
\item 1 \cmd{hello} would be launched on {\tt n0} (named
|
|
\host{node1})
|
|
\item 1 \cmd{hello} would be launched on {\tt n1} (named
|
|
\host{node2})
|
|
\item 2 \cmd{hello}s would be launched on {\tt n2} (named
|
|
\host{node3})
|
|
\end{itemize}
|
|
|
|
Note that any number can be used -- if a number is used that is
|
|
greater than how many CPUs are in the LAM universe, LAM will ``wrap
|
|
around'' and start scheduling starting with the first node again. For
|
|
example, using \cmdarg{-np 10} would result in the following
|
|
schedule:
|
|
|
|
\begin{itemize}
|
|
\item 2 \cmd{hello}s on {\tt n0} (1 from the first pass, and then a
|
|
second from the ``wrap around'')
|
|
\item 2 \cmd{hello}s on {\tt n1} (1 from the first pass, and then a
|
|
second from the ``wrap around'')
|
|
\item 4 \cmd{hello}s on {\tt n2} (2 from the first pass, and then 2
|
|
more from the ``wrap around'')
|
|
\item 2 \cmd{hello}s on {\tt n3}
|
|
\end{itemize}
|
|
|
|
The \file{mpirun(1)} man page contains much more information and
|
|
\cmd{mpirun} and the options available. For example, \cmd{mpirun}
|
|
also supports Multiple Program, Multiple Data (MPMD) programs,
|
|
although it is not discussed here. Also see
|
|
Section~\ref{sec:commands-mpirun} (page~\pageref{sec:commands-mpirun})
|
|
in this document.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{The \icmd{mpiexec} Command}
|
|
|
|
The MPI-2 standard recommends the use of \cmd{mpiexec} for portable
|
|
MPI process startup. In Open MPI, \cmd{mpiexec} is functionaly similar
|
|
to \cmd{mpirun}. Some options that are available to \cmd{mpirun} are
|
|
not available to \cmd{mpiexec}, and vice-versa. The end result is
|
|
typically the same, however -- both will launch parallel MPI programs;
|
|
which you should use is likely simply a personal choice.
|
|
|
|
That being said, \cmd{mpiexec} offers more convenient access in three
|
|
cases:
|
|
|
|
\begin{itemize}
|
|
\item Running MPMD programs
|
|
\item Running heterogeneous programs
|
|
\item Running ``one-shot'' MPI programs (i.e., boot LAM, run the
|
|
program, then halt LAM)
|
|
\end{itemize}
|
|
|
|
The general syntax for \cmd{mpiexec} is:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpiexec <global_options> <cmd1> : <cmd2> : ...
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
%%%%%
|
|
|
|
\subsubsection{Running MPMD Programs}
|
|
|
|
For example, to run a manager/worker parallel program, where two
|
|
different executables need to be launched (i.e., \cmd{manager} and
|
|
\cmd{worker}, the following can be used:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpiexec -n 1 manager : worker
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
This runs one copy of \cmd{manager} and one copy of \cmd{worker} for
|
|
every CPU in the LAM universe.
|
|
|
|
%%%%%
|
|
|
|
\subsubsection{Running Heterogeneous Programs}
|
|
|
|
Since LAM is a heterogeneous MPI implementation, it supports running
|
|
heterogeneous MPI programs. For example, this allows running a
|
|
parallel job that spans a Sun SPARC machine and an IA-32 Linux machine
|
|
(even though they are opposite endian machines). Although this can be
|
|
somewhat complicated to setup (remember that you will first need to
|
|
\cmd{lamboot} successfully, which essentially means that LAM must be
|
|
correctly installed on both architectures), the \cmd{mpiexec} command
|
|
can be helpful in actually running the resulting MPI job.
|
|
|
|
Note that you will need to have two MPI executables -- one compiled
|
|
for Solaris (e.g., \cmd{hello.solaris}) and one compiled for Linux
|
|
(e.g., \cmd{hello.linux}). Assuming that these executables both
|
|
reside in the same directory, and that directory is available on both
|
|
nodes (or the executables can be found in the \envvar{PATH} on their
|
|
respective machines), the following command can be used:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpiexec -arch solaris hello.solaris : -arch linux hello.linux
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
This runs the \cmd{hello.solaris} command on all nodes in the LAM
|
|
universe that have the string ``solaris'' anywhere in their
|
|
architecture string, and \cmd{hello.linux} on all nodes that have
|
|
``linux'' in their architecture string. The architecture string of a
|
|
given LAM installation can be found by running the \cmd{laminfo}
|
|
command.
|
|
|
|
%%%%%
|
|
|
|
\subsubsection{``One-Shot'' MPI Programs}
|
|
|
|
In some cases, it seems like extra work to boot a LAM universe, run
|
|
a single MPI job, and then shut down the universe. Batch jobs are
|
|
good examples of this -- since only one job is going to be run, why
|
|
does it take three commands? \cmd{mpiexec} provides a convenient way
|
|
to run ``one-shot'' MPI jobs.
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ mpiexec -machinefile hostfile hello
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
This will invoke \cmd{lamboot} with the boot schema named
|
|
``\file{hostfile}'', run the MPI program \cmd{hello} on all available
|
|
CPUs in the resulting universe, and then shut down the universe with
|
|
the \cmd{lamhalt} command (which we'll discuss in
|
|
Section~\ref{sec:getting-started-lamhalt}, below).
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{The \icmd{mpitask} Command}
|
|
|
|
The \cmd{mpitask} command is analogous to the sequential Unix command
|
|
\cmd{ps}. It shows the current status of the MPI program(s) being
|
|
executed in the LAM universe, and displays primitive information about
|
|
what MPI function each process is currently executing (if any). Note
|
|
that in normal practice, the \cmd{mpimsg} command only gives a
|
|
snapshot of what messages are flowing between MPI processes, and
|
|
therefore is usually only accurate at that single point in time. To
|
|
really debug message passing traffic, use a tool such as message
|
|
passing analyzer (e.g., XMPI), or a parallel debugger (e.g.,
|
|
TotalView).
|
|
|
|
\cmd{mpitask} can be run from any node in the LAM universe.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\subsection{The \icmd{lamclean} Command}
|
|
|
|
The \cmd{lamclean} command completely removed all running programs
|
|
from the LAM universe. This can be useful if a parallel job crashes
|
|
and/or leaves state in the LAM run-time environment (e.g., MPI-2
|
|
published names). It is usually run with no parameters:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ lamclean
|
|
\end{lstlisting}
|
|
% stupid emacs mode: $
|
|
|
|
\cmd{lamclean} is typically only necessary when developing / debugging
|
|
MPI applications -- i.e., programs that hang, messages that are left
|
|
around, etc. Correct MPI programs should terminate properly, clean up
|
|
all their messages, unpublish MPI-2 names, etc.
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\section{Shutting Down the LAM Universe}
|
|
\label{sec:getting-started-lamhalt}
|
|
|
|
When finished with the LAM universe, it should be shut down with the
|
|
\icmd{lamhalt} command:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ lamhalt
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
In most cases, this is sufficient to kill all running MPI processes
|
|
and shut down the LAM universe.
|
|
|
|
However, in some rare conditions, \cmd{lamhalt} may fail. For
|
|
example, if any of the nodes in the LAM universe crashed before
|
|
running \cmd{lamhalt}, \cmd{lamhalt} will likely timeout and
|
|
potentially not kill the entire LAM universe. In this case, you will
|
|
need to use the \icmd{lamwipe} command to guarantee that the LAM
|
|
universe has shut down properly:
|
|
|
|
\lstset{style=lam-cmdline}
|
|
\begin{lstlisting}
|
|
shell$ lamwipe -v hostfile
|
|
\end{lstlisting}
|
|
% Stupid emacs mode: $
|
|
|
|
\noindent where \file{hostfile} is the same boot schema that was used to
|
|
boot LAM (i.e., all the same nodes are listed). \cmd{lamwipe} will
|
|
forcibly kill all Open MPI processes and terminate the LAM universe.
|
|
This is a slower process than \cmd{lamhalt}, and is typically not
|
|
necessary.
|
|
|