openmpi/contrib/build-mca-comps-outside-of-tree/README.md

# Description

2 Feb 2011

This sample `tcp2` BTL component is a simple example of how to build
an Open MPI MCA component from outside of the Open MPI source tree.
This is a valuable technique for 3rd parties who want to provide their
own components for Open MPI, but do not want to be in the mainstream
distribution (i.e., their code is not part of the main Open MPI code
base).

Previous colloquial knowledge held that building a component from
outside of the Open MPI source tree required configuring Open MPI
`--with-devel-headers`, and then building and installing it.  This
configure switch installs all of OMPI's internal `.h` files under
`$prefix/include/openmpi`, and therefore allows 3rd party code to be
compiled outside of the Open MPI tree.

This method definitely works, but is annoying:

* You have to ask users to use this special configure switch.
* Not all users install from source; many get binary packages (e.g.,
  RPMs).

This example package shows two ways to build an Open MPI MCA component
from outside the Open MPI source tree:

1. Using the above `--with-devel-headers` technique
2. Compiling against the Open MPI source tree itself (vs. the
   installation tree)

The user still has to have a source tree, but at least they don't have
to be required to use `--with-devel-headers` (which most users don't) --
they can likely build off the source tree that they already used.

# Example project contents

The `tcp2` component is a direct copy of the TCP BTL as of January
2011 -- it has just been renamed so that it can be built separately
and installed alongside the real TCP BTL component.

Most of the mojo for both methods is handled in the example
components' `configure.ac`, but the same techniques are applicable
outside of the GNU Auto toolchain.

This sample `tcp2` component has an `autogen.sh` script that requires
the normal Autoconf, Automake, and Libtool.  It also adds the
following two configure switches:

1. `--with-openmpi-install=DIR`:
    If provided, `DIR` is an Open MPI installation tree that was
    installed `--with-devel-headers`.

    This switch uses the installed `mpicc --showme:<foo>` functionality
    to extract the relevant `CPPFLAGS`, `LDFLAGS`, and `LIBS`.
1. `--with-openmpi-source=DIR`:
    If provided, `DIR` is the source of a configured and built Open MPI
    source tree (corresponding to the version expected by the example
    component).  The source tree is not required to have been
    configured `--with-devel-headers`.

    This switch uses the source tree's `config.status` script to
    extract the relevant `CPPFLAGS` and `CFLAGS`.

Either one of these two switches must be provided, or appropriate
`CPPFLAGS`, `CFLAGS`, `LDFLAGS`, and/or `LIBS` must be provided such
that valid Open MPI header and library files can be found and compiled
/ linked against, respectively.

# Example use

First, download, build, and install Open MPI:

```
$ cd $HOME
$ wget https://www.open-mpi.org/software/ompi/vX.Y/downloads/openmpi-X.Y.Z.tar.bz2
[...lots of output...]
$ tar jxf openmpi-X.Y.Z.tar.bz2
$ cd openmpi-X.Y.Z
$ ./configure --prefix=/opt/openmpi ...
[...lots of output...]
$ make -j 4 install
[...lots of output...]
$ /opt/openmpi/bin/ompi_info | grep btl
                 MCA btl: self (MCA vA.B, API vM.N, Component vX.Y.Z)
                 MCA btl: sm (MCA vA.B, API vM.N, Component vX.Y.Z)
                 MCA btl: tcp (MCA vA.B, API vM.N, Component vX.Y.Z)
  [where X.Y.Z, A.B, and M.N are appropriate for your version of Open MPI]
$
```

Notice the installed BTLs from `ompi_info`.

Now `cd` into this example project and build it, pointing it to the
source directory of the Open MPI that you just built.  Note that we
use the same `--prefix` as when installing Open MPI (so that the built
component will be installed into the Right place):

```
$ cd /path/to/this/sample
$ ./autogen.sh
$ ./configure --prefix=/opt/openmpi --with-openmpi-source=$HOME/openmpi-X.Y.Z
[...lots of output...]
$ make -j 4 install
[...lots of output...]
$ /opt/openmpi/bin/ompi_info | grep btl
                 MCA btl: self (MCA vA.B, API vM.N, Component vX.Y.Z)
                 MCA btl: sm (MCA vA.B, API vM.N, Component vX.Y.Z)
                 MCA btl: tcp (MCA vA.B, API vM.N, Component vX.Y.Z)
                 MCA btl: tcp2 (MCA vA.B, API vM.N, Component vX.Y.Z)
  [where X.Y.Z, A.B, and M.N are appropriate for your version of Open MPI]
$
```

Notice that the `tcp2` BTL is now installed.

# Random notes

The component in this project is just an example; I whipped it up in
the span of several hours.  Your component may be a bit more complex
than this or have slightly different requirements.  So you may need to
tweak the configury or build system in each of the components to fit
what you need.

Changes required to the component to make it build in a standalone
mode:

1. Write your own `configure` script.  This component is just a
   sample.  You basically need to build against an OMPI install that
   was installed `--with-devel-headers` or a built OMPI source tree.
   See `./configure --help` for details.
1. I also provided a bogus `btl_tcp2_config.h` (generated by
   `configure`).  This file is not included anywhere, but it does
   provide protection against re-defined `PACKAGE_*` macros when
   running `configure`, which is quite annoying.
1. Modify `Makefile.am` to only build DSOs.  I.e., you can optionally
   take the static option out since the component can *only* build in
   DSO mode when building standalone.  That being said, it doesn't
   hurt to leave the static builds in -- this would (hypothetically)
   allow the component to be built both in-tree and out-of-tree.

Ping the Open MPI devel list if you have questions about this
project.

Enjoy.

- Jeff Squyres