From eb7365581adb0d26f1f6e1b89668a136119269d7 Mon Sep 17 00:00:00 2001 From: Jeff Squyres Date: Fri, 5 Aug 2005 12:21:32 +0000 Subject: [PATCH] Copy over from the beta branch. This commit was SVN r6745. --- README | 488 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 485 insertions(+), 3 deletions(-) diff --git a/README b/README index b92381794c..e235a9ffae 100644 --- a/README +++ b/README @@ -12,7 +12,489 @@ Additional copyrights may follow $HEADER$ -This is the world's greatest MPI. +The Open MPI development team is pleased to announce an beta-quality +release of the current state of our software to the public. The +software is "mostly stable", but is definitely rough around the edges +-- it is not complete and not completely debugged. Our intent with +this release is to get the Open MPI software onto other people's +machines and see what creative ways you can come up with to break it, +and to send us your comments and suggestions. -...more to write here -should be a short file, referring to the real docs. +However, we want to stress the following point: this is an *beta* +quality release. We're quite aware that there are several things +still broken (but report them anyway!). + +Also note that many performance enhancements were applied to the Open +MPI code repository *after* we forked for this beta release (in +mid-May, 2005). To get the most recent advancements, you'll need to +get a checkout of the current code base (keeping in mind that that is +the head of development, and is not always stable!). See +http://www.open-mpi.org/svn/ for details. + +The best way to report bugs, send comments, or ask questions is to +sign up on the user's and/or developer's mailing list (for user-level +and developer-level questions; when in doubt, send to the user's +list): + + users@open-mpi.org + devel@open-mpi.org + +Because of spam, only subscribers are allowed to post to these lists +(ensure that you subscribe with and post from exactly the same e-mail +address -- joe@example.com is considered different than +joe@mycomputer.example.com!). Visit these pages to subscribe to the +lists: + + http://www.open-mpi.org/mailman/listinfo.cgi/users + http://www.open-mpi.org/mailman/listinfo.cgi/devel + +Thanks for your time. + +=========================================================================== + +This tarball is a beta release of Open MPI, representing the head of +development as of mid-May 2005. It is not yet complete. + +- All MPI point-to-point transports were extensively tuned for + performance after branching for this beta. As result, some + point-to-point transports in this beta do not perform well, + especially for large messages (e.g,. shared memory). + +- The Open MPI installation must be in your PATH on all nodes (and + potentially LD_LIBRARY_PATH, if libmpi is a shared library). + +- LAM/MPI-like mpirun notation of "C" and "N" is not yet supported. + +- Support for Quadrics and Infiniband (both mVAPI and OpenIB) is + missing (see the current code base). + +- Shared memory support will not function properly on machines that + have a weak memory consistency mode. The default in this beta is to + disable shared memory support on all Power PC architectures, even + though some Power PC platforms have strong memory consistency + models. See the description of the --enable-ptl-sm configure flag, + below. + +- Striping MPI messages across multiple networks is supported (and + happens automatically when multiple networks are available), but + needs performance tuning. + +- The only run-time systems currently supported are: + - rsh / ssh + - BProc of the flavor that is used at Los Alamos National Labs (in + particular, it must be used with the BJS scheduler) + +- Complete user and system administrator documentation is missing + (this file comprises the majority of the current user + documentation). + +- Missing MPI functionality: + - The Fortran 90 MPI API is disabled (it is not complete). + - MPI-2 one-sided functionality will not be included in the first + few releases of Open MPI. + +- Systems that have been tested are: + - Linux, 32 bit, with gcc + - OS X (10.3), 32 bit, with gcc + - OS X (10.4), 32 bit, with gcc + +- Other systems have been lightly (but not fully tested): + - Other compilers on Linux, 32 bit + - 64 bit platforms (AMD, PPC64, Sparc); they "mostly work", but + there are still some known issues + +- As of 9 June 2005, Open MPI will not function on OS X Tiger (10.4) + in 64 bit mode due to an ioctl() bug in OS X. Apple has been + notified of the problem. + +- There are some cases where after running MPI applications, the + directory /tmp/openmpi-sessions-@* will exist + (but will likely be empty). It is safe to remove after the run is + complete. + +- The MPI and run-time layers do not free all used memory properly + during MPI_FINALIZE. + +- Running on nodes with different endian and/or different datatype + sizes within a single parallel application is not supported in this + beta. + +- Threading support (both asynchronous progress and + MPI_THREAD_MULTIPLE) is included, but is only lightly tested. + + +=========================================================================== + +Building Open MPI +----------------- + +Open MPI uses a traditional configure script paired with "make" to +build. Typical installs can be of the pattern: + +--------------------------------------------------------------------------- +shell$ ./configure [...options...] +shell$ make all install +--------------------------------------------------------------------------- + +There are many available configure options; a summary of the more +important ones follows: + +--prefix= + Install Open MPI into the base directory named . Hence, + Open MPI will place its executables in /bin, its header + files in /include, its libraries in /lib, etc. + +--with-ptl-gm= + Specify the directory where the GM libraries and header files are + located. This enables GM support in Open MPI. + +--with-mpi-param_check(=value) + "value" can be one of: always, never, runtime. If no value is + specified, or this option is not used, "always" is the default. + Using --without-mpi-param-check is equivalent to "never". + - always: the parameters of MPI functions are always checked for + errors + - never: the parameters of MPI functions are never checked for + errors + - runtime: whether the parameters of MPI functions are checked + depends on the value of the MCA parameter mpi_param_check + (default: yes). + +--with-threads=value + Since thread support (both support for MPI_THREAD_MULTIPLE and + asynchronous progress) is only partially tested, it is disabled by + default. To enable threading, use "--with-threads=posix". This is + most useful when combined with --enable-mpi-threads and/or + --enable-progress-threads. + +--enable-mpi-threads + Allows the MPI thread level MPI_THREAD_MULTIPLE. See + --with-threads; this is currently disabled by default. + +--enable-progress-threads + Allows asynchronous progress in some transports. See + --with-threads; this is currently disabled by default. + +--disable-shared + By default, libmpi is built as a shared library, and all components + are built as dynamic shared objects (DSOs). This switch disables + this default; it is really only useful when used with + --enable-static. + +--enable-static + Build libmpi as a static library, and statically link in all + components. + +--enable-ptl-sm + This beta does not support shared memory on platforms with weak + memory consistency models. As such, the default in this beta is to + disable shared memory support for all Power PC platforms. However, + some Power PC platforms have strong memory consistency models (e.g., + G4 and POWER3). On these platforms, shared memory support can be + enabled with the --enable-ptl-sm flag. It is strongly discouraged + to enable shared memory support on other Power PC platforms (e.g., + G5, POWER4). + +There are other options available -- see "./configure --help". + +Open MPI supports all the "make" targets that are provided by GNU +Automake, such as: + +all - build the entire package +install - install the package +uninstall - remove all traces of the package from the $prefix +clean - clean out the build tree + +Once Open MPI has been built and installed, it is safe to run "make +clean" and/or remove the entire build tree. + +VPATH builds are fully supported. + +Generally speaking, the only thing that users need to do to use Open +MPI is ensure that /bin is in their PATH. Users may need to +ensure that this directory is set in their PATH in their shell setup +files (e.g., .bashrc, .cshrc) so that rsh/ssh-based logins will be +able to find the Open MPI executables. + +Setting LD_LIBRARY_PATH is typically not necessary, but in some cases, +if libmpi.so cannot be found when MPI applications are run, +/lib should be added to LD_LIBRARY_PATH. + +=========================================================================== + +Checking Your Open MPI Installation +----------------------------------- + +The "ompi_info" command can be used to check the status of your Open +MPI installation (located in /bin/ompi_info). Running it with +no arguments provides a summary of information about your Open MPI +installation. + +Note that the ompi_info command is extremely helpful in determining +which components are installed as well as listing all the run-time +settable parameters that are available in each component (as well as +their default values). + +The following options may be helpful: + +--all Show a *lot* of information about your Open MPI + installation. +--parsable Display all the information in an easily + grep/cut/awk/sed-able format. +--param + A of "all" and a of "all" will + show all parameters to all components. Otherwise, the + parameters of all the components in a specific framework, + or just the parameters of a specific component can be + displayed by using an appropriate and/or + name. + +Changing the values of these parameters is explained in the "The +Modular Component Architecture (MCA)" section, below. + +=========================================================================== + +Compiling Open MPI Applications +------------------------------- + +Open MPI provides "wrapper" compilers that should be used for +compiling MPI applications: + +C: mpicc +C++: mpiCC (or mpic++ if your filesystem is case-insensitive) +Fortran 77: mpif77 +Fortran 90: mpif90 + +For example: + +shell$ mpicc hello_world_mpi.c -o hello_world_mpi -g +shell$ + +All the wrapper compilers do is add a variety of compiler and linker +flags to the command line and then invoke a back-end compiler. The +end result is an MPI executable that is properly linked to all the +relevant libraries. + +=========================================================================== + +Running Open MPI Applications +----------------------------- + +Open MPI supports both mpirun and mpiexec (they are actually the +same). For example: + +shell$ mpirun -np 2 hello_world_mpi + +or + +shell$ mpiexec -np 1 hello_world_mpi : -np 1 hello_world_mpi + +are equivalent. Many of mpiexec's switches (such as -host and -arch) +are not yet functional, although they will not error if you try to use +them. + +Since rsh is probably the launcher that you will be using (if you are +outside of Los Alamos National Laboratory), you can also specify a +-hostfile parameter, indicating an standard mpirun-style hostfile (one +hostname per line): + +shell$ mpirun -hostfile my_hostfile -np 2 hello_world_mpi + +If you intend to run more than one process on a node, the hostfile can +use the "slots" attribute. If "slots" is not specified, a count of 1 +is assumed. For example, using the following hostfile: + +--------------------------------------------------------------------------- +node1.example.com +node2.example.com +node3.example.com slots=2 +node4.example.com slots=4 +--------------------------------------------------------------------------- + +shell$ mpirun -hostfile my_hostfile -np 8 hello_world_mpi + +will launch MPI_COMM_WORLD rank 0 on node1, rank 1 on node2, ranks 2 +and 3 on node3, and ranks 4 through 7 on node4. + +Note that the values of component parameters can be changed on the +mpirun / mpiexec command line. This is explained in the section +below, "The Modular Component Architecture (MCA)". + +=========================================================================== + +The Modular Component Architecture (MCA) + +The MCA is the backbone of Open MPI -- most services and functionality +are implemented through MCA components. Here is a list of all the +component frameworks in Open MPI: + +--------------------------------------------------------------------------- +MPI component frameworks: +------------------------- + +coll - MPI collective algorithms +io - MPI-2 I/O +pml - MPI point-to-point management layer +ptl - MPI point-to-point transport layer +topo - MPI topology routines + +Back-end run-time environment component frameworks: +--------------------------------------------------- + +errmgr - RTE error manager +gpr - General purpose registry +iof - I/O forwarding +ns - Name server +oob - Out of band messaging +pls - Process launch system +ras - Resource allocation system +rds - Resource discovery system +rmaps - Resource mapping system +rmgr - Resource manager +rml - RTE message layer +soh - State of health monitor + +Miscellaneous frameworks: +------------------------- + +allocator - Memory allocator +mpool - Memory pooling +--------------------------------------------------------------------------- + +Each framework typically has one or more components that are used at +run-time. For example, the ptl framework is used by MPI to send bytes +across underlying networks. The tcp ptl, for example, sends messages +across TCP-based networks; the gm ptl sends messages across GM +Myrinet-based networks. + +Each component typically has some tunable parameters that can be +changed at run-time. Use the ompi_info command to check a component +to see what its tunable parameters are. For example: + +shell$ ompi_info --param ptl tcp + +shows all the parameters (and default values) for the tcp ptl +component. + +These values can be overridden at run-time in several ways. At +run-time, the following locations are examined (in order) for new +values of parameters: + +1. /etc/openmpi-mca-params.conf + + This file is intended to set any system-wide default MCA parameter + values -- it will apply, by default, to all users who use this Open + MPI installation. The default file that is installed contains many + comments explaining its format. + +2. $HOME/.openmpi/mca-params.conf + + If this file exists, it should be in the same format as + /etc/openmpi-mca-params.conf. It is intended to provide + per-user default parameter values. + +3. environment variables of the form OMPI_MCA_ set equal to a + + + Where is the name of the parameter. For example, set the + variable named OMPI_MCA_ptl_tcp_frag_size to the value 65536 + (Bourne-style shells): + + shell$ OMPI_MCA_ptl_tcp_frag_size=65536 + shell$ export OMPI_MCA_ptl_tcp_frag_size + +4. the mpirun command line: --mca + + Where is the name of the parameter. For example: + + shell$ mpirun --mca ptl_tcp_frag_size 65536 -np 2 hello_world_mpi + +These locations are checked in order. For example, a parameter value +passed on the mpirun command line will override an environment +variable; an environment variable will override the system-wide +defaults. + +=========================================================================== + +Common Questions +---------------- + +1. How do I change the rsh/ssh launcher to use rsh? + + The default remote shell agent for the rsh/ssh launcher is ssh, but + you can set an MCA parameter to change it to rsh (or use a specific + path for ssh, pass different parameters to rsh/ssh, etc.). The MCA + parameter name is pls_rsh_agent. You can use any of the methods + for setting MCA parameters described above; for example: + + shell$ mpirun --mca pls_rsh_agent rsh -np 4 a.out + +...we'll add more questions here as they are asked by real users. + +=========================================================================== + +Got more questions? +------------------- + +Found a bug? Got a question? Want to make a suggestion? Want to +contribute to Open MPI? Please let us know! + +User-level questions and comments should generally be sent to the +user's mailing list (users@open-mpi.org). Because of spam, only +subscribers are allowed to post to this list (ensure that you +subscribe with and post from *exactly* the same e-mail address -- +joe@example.com is considered different than +joe@mycomputer.example.com!). Visit this page to subscribe to the +user's list: + + http://www.open-mpi.org/mailman/listinfo.cgi/users + +Developer-level bug reports, questions, and comments should generally +be sent to the developer's mailing list (devel@open-mpi.org). Please +do not post the same question to both lists. As with the user's list, +only subscribers are allowed to post to the developer's list. Visit +the following web page to subscribe: + + http://www.open-mpi.org/mailman/listinfo.cgi/devel + +When submitting bug reports to either list, be sure to include the +following information in your mail (please compress!): + +- the stdout and stderr from Open MPI's configure +- the top-level config.log file +- the stdout and stderr from building Open MPI +- the output from "ompi_info --all" (if possible) + +For Bourne-type shells, here's one way to capture this information: + +shell$ ./configure ... 2>&1 | tee config.out +[...lots of configure output...] +shell$ make 2>&1 | tee make.out +[...lots of make output...] +shell$ mkdir ompi-output +shell$ cp config.out config.log make.out ompi-output +shell$ ompi_info --all |& tee ompi-output/ompi-info.out +shell$ tar cvf ompi-output.tar ompi-output +[...output from tar...] +shell$ gzip ompi-output.tar + +For C shell-type shells, the procedure is only slightly different: + +shell% ./configure ... |& tee config.out +[...lots of configure output...] +shell% make |& tee make.out +[...lots of make output...] +shell% mkdir ompi-output +shell% cp config.out config.log make.out ompi-output +shell% ompi_info --all |& tee ompi-output/ompi-info.out +shell% tar cvf ompi-output.tar ompi-output +[...output from tar...] +shell% gzip ompi-output.tar + +In either case, attach the resulting ompi-output.tar.gz file to your +mail. This provides the Open MPI developers with a lot of information +about your installation and can greatly assist us in helping with your +problem. + +Be sure to also include any other useful files (in the +ompi-output.tar.gz tarball), such as output showing specific errors.