Two major things in this commit:
* New "op" MPI layer framework
* Addition of the MPI_REDUCE_LOCAL proposed function (for MPI-2.2)
= Op framework =
Add new "op" framework in the ompi layer. This framework replaces the
hard-coded MPI_Op back-end functions for (MPI_Op, MPI_Datatype) tuples
for pre-defined MPI_Ops, allowing components and modules to provide
the back-end functions. The intent is that components can be written
to take advantage of hardware acceleration (GPU, FPGA, specialized CPU
instructions, etc.). Similar to other frameworks, components are
intended to be able to discover at run-time if they can be used, and
if so, elect themselves to be selected (or disqualify themselves from
selection if they cannot run). If specialized hardware is not
available, there is a default set of functions that will automatically
be used.
This framework is ''not'' used for user-defined MPI_Ops.
The new op framework is similar to the existing coll framework, in
that the final set of function pointers that are used on any given
intrinsic MPI_Op can be a mixed bag of function pointers, potentially
coming from multiple different op modules. This allows for hardware
that only supports some of the operations, not all of them (e.g., a
GPU that only supports single-precision operations).
All the hard-coded back-end MPI_Op functions for (MPI_Op,
MPI_Datatype) tuples still exist, but unlike coll, they're in the
framework base (vs. being in a separate "basic" component) and are
automatically used if no component is found at runtime that provides a
module with the necessary function pointers.
There is an "example" op component that will hopefully be useful to
those writing meaningful op components. It is currently
.ompi_ignore'd so that it doesn't impinge on other developers (it's
somewhat chatty in terms of opal_output() so that you can tell when
its functions have been invoked). See the README file in the example
op component directory. Developers of new op components are
encouraged to look at the following wiki pages:
https://svn.open-mpi.org/trac/ompi/wiki/devel/Autogen
https://svn.open-mpi.org/trac/ompi/wiki/devel/CreateComponent
https://svn.open-mpi.org/trac/ompi/wiki/devel/CreateFramework
= MPI_REDUCE_LOCAL =
Part of the MPI-2.2 proposal listed here:
https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/24
is to add a new function named MPI_REDUCE_LOCAL. It is very easy to
implement, so I added it (also because it makes testing the op
framework pretty easy -- you can do it in serial rather than via
parallel reductions). There's even a man page!
This commit was SVN r20280.
2009-01-14 23:44:31 +00:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
|
|
|
* University Research and Technology
|
|
|
|
* Corporation. All rights reserved.
|
|
|
|
* Copyright (c) 2004-2005 The University of Tennessee and The University
|
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
* reserved.
|
|
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
|
|
* University of Stuttgart. All rights reserved.
|
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
* All rights reserved.
|
|
|
|
* Copyright (c) 2008-2009 Cisco Systems, Inc. All rights reserved.
|
|
|
|
* $COPYRIGHT$
|
|
|
|
*
|
|
|
|
* Additional copyrights may follow
|
|
|
|
*
|
|
|
|
* $HEADER$
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include "ompi_config.h"
|
|
|
|
|
|
|
|
#include <stdio.h>
|
|
|
|
#include <stdlib.h>
|
|
|
|
#include <string.h>
|
|
|
|
|
|
|
|
#include "mpi.h"
|
|
|
|
|
|
|
|
#include "opal/class/opal_list.h"
|
2009-02-14 02:26:12 +00:00
|
|
|
#include "opal/util/output.h"
|
Two major things in this commit:
* New "op" MPI layer framework
* Addition of the MPI_REDUCE_LOCAL proposed function (for MPI-2.2)
= Op framework =
Add new "op" framework in the ompi layer. This framework replaces the
hard-coded MPI_Op back-end functions for (MPI_Op, MPI_Datatype) tuples
for pre-defined MPI_Ops, allowing components and modules to provide
the back-end functions. The intent is that components can be written
to take advantage of hardware acceleration (GPU, FPGA, specialized CPU
instructions, etc.). Similar to other frameworks, components are
intended to be able to discover at run-time if they can be used, and
if so, elect themselves to be selected (or disqualify themselves from
selection if they cannot run). If specialized hardware is not
available, there is a default set of functions that will automatically
be used.
This framework is ''not'' used for user-defined MPI_Ops.
The new op framework is similar to the existing coll framework, in
that the final set of function pointers that are used on any given
intrinsic MPI_Op can be a mixed bag of function pointers, potentially
coming from multiple different op modules. This allows for hardware
that only supports some of the operations, not all of them (e.g., a
GPU that only supports single-precision operations).
All the hard-coded back-end MPI_Op functions for (MPI_Op,
MPI_Datatype) tuples still exist, but unlike coll, they're in the
framework base (vs. being in a separate "basic" component) and are
automatically used if no component is found at runtime that provides a
module with the necessary function pointers.
There is an "example" op component that will hopefully be useful to
those writing meaningful op components. It is currently
.ompi_ignore'd so that it doesn't impinge on other developers (it's
somewhat chatty in terms of opal_output() so that you can tell when
its functions have been invoked). See the README file in the example
op component directory. Developers of new op components are
encouraged to look at the following wiki pages:
https://svn.open-mpi.org/trac/ompi/wiki/devel/Autogen
https://svn.open-mpi.org/trac/ompi/wiki/devel/CreateComponent
https://svn.open-mpi.org/trac/ompi/wiki/devel/CreateFramework
= MPI_REDUCE_LOCAL =
Part of the MPI-2.2 proposal listed here:
https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/24
is to add a new function named MPI_REDUCE_LOCAL. It is very easy to
implement, so I added it (also because it makes testing the op
framework pretty easy -- you can do it in serial rather than via
parallel reductions). There's even a man page!
This commit was SVN r20280.
2009-01-14 23:44:31 +00:00
|
|
|
#include "opal/mca/mca.h"
|
|
|
|
#include "opal/mca/base/base.h"
|
|
|
|
#include "opal/mca/base/mca_base_component_repository.h"
|
|
|
|
|
|
|
|
|
|
|
|
#include "ompi/constants.h"
|
|
|
|
#include "ompi/mca/op/op.h"
|
|
|
|
#include "ompi/mca/op/base/base.h"
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Global variables
|
|
|
|
*/
|
|
|
|
bool ompi_op_base_components_available_valid = false;
|
|
|
|
opal_list_t ompi_op_base_components_available;
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Private functions
|
|
|
|
*/
|
|
|
|
static int init_query(const mca_base_component_t * ls,
|
|
|
|
mca_base_component_priority_list_item_t * entry,
|
|
|
|
bool enable_progress_threads,
|
|
|
|
bool enable_mpi_threads);
|
|
|
|
static int init_query_1_0_0(const mca_base_component_t * ls,
|
|
|
|
mca_base_component_priority_list_item_t *
|
|
|
|
entry, bool enable_progress_threads,
|
|
|
|
bool enable_mpi_threads);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Scan down the list of successfully opened components and query each
|
|
|
|
* of them (the opened list will be one or more components. If the
|
|
|
|
* user requested a specific set of components, they will be the only
|
|
|
|
* components in the opened list). Create and populate the available
|
|
|
|
* list of all components who indicate that they want to be considered
|
|
|
|
* for selection. Close all components who do not want to be
|
|
|
|
* considered for selection. Finally, destroy the "opened" list,
|
|
|
|
* because the only the "available" list is relevant now.
|
|
|
|
*/
|
|
|
|
int ompi_op_base_find_available(bool enable_progress_threads,
|
|
|
|
bool enable_mpi_threads)
|
|
|
|
{
|
|
|
|
bool found = false;
|
|
|
|
mca_base_component_priority_list_item_t *entry;
|
|
|
|
opal_list_item_t *p;
|
|
|
|
const mca_base_component_t *component;
|
|
|
|
|
|
|
|
/* Initialize the list */
|
|
|
|
|
|
|
|
OBJ_CONSTRUCT(&ompi_op_base_components_available, opal_list_t);
|
|
|
|
ompi_op_base_components_available_valid = true;
|
|
|
|
|
|
|
|
/* The list of components that we should check has already been
|
|
|
|
established in ompi_op_base_open. */
|
|
|
|
|
|
|
|
for (found = false,
|
|
|
|
p = opal_list_remove_first(&ompi_op_base_components_opened);
|
|
|
|
p != NULL;
|
|
|
|
p = opal_list_remove_first(&ompi_op_base_components_opened)) {
|
|
|
|
component = ((mca_base_component_list_item_t *) p)->cli_component;
|
|
|
|
|
|
|
|
/* Call a subroutine to do the work, because the component may
|
|
|
|
represent different versions of the op MCA. */
|
|
|
|
|
|
|
|
entry = OBJ_NEW(mca_base_component_priority_list_item_t);
|
|
|
|
entry->super.cli_component = component;
|
|
|
|
entry->cpli_priority = 0;
|
|
|
|
if (OMPI_SUCCESS == init_query(component, entry,
|
|
|
|
enable_progress_threads,
|
|
|
|
enable_mpi_threads)) {
|
|
|
|
opal_list_append(&ompi_op_base_components_available,
|
|
|
|
(opal_list_item_t *) entry);
|
|
|
|
found = true;
|
|
|
|
} else {
|
|
|
|
|
|
|
|
/* If the component doesn't want to run, then close it.
|
|
|
|
It's already had its close() method invoked; now close
|
|
|
|
it out of the DSO repository (if it's there). */
|
|
|
|
|
|
|
|
mca_base_component_repository_release(component);
|
|
|
|
OBJ_RELEASE(entry);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Free the entry from the "opened" list */
|
|
|
|
|
|
|
|
OBJ_RELEASE(p);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* The opened list is now no longer useful and we can free it */
|
|
|
|
|
|
|
|
OBJ_DESTRUCT(&ompi_op_base_components_opened);
|
|
|
|
ompi_op_base_components_opened_valid = false;
|
|
|
|
|
2009-01-15 02:01:32 +00:00
|
|
|
/* It is not an error if there are no components available; we'll
|
|
|
|
just fall back to the base functions. */
|
Two major things in this commit:
* New "op" MPI layer framework
* Addition of the MPI_REDUCE_LOCAL proposed function (for MPI-2.2)
= Op framework =
Add new "op" framework in the ompi layer. This framework replaces the
hard-coded MPI_Op back-end functions for (MPI_Op, MPI_Datatype) tuples
for pre-defined MPI_Ops, allowing components and modules to provide
the back-end functions. The intent is that components can be written
to take advantage of hardware acceleration (GPU, FPGA, specialized CPU
instructions, etc.). Similar to other frameworks, components are
intended to be able to discover at run-time if they can be used, and
if so, elect themselves to be selected (or disqualify themselves from
selection if they cannot run). If specialized hardware is not
available, there is a default set of functions that will automatically
be used.
This framework is ''not'' used for user-defined MPI_Ops.
The new op framework is similar to the existing coll framework, in
that the final set of function pointers that are used on any given
intrinsic MPI_Op can be a mixed bag of function pointers, potentially
coming from multiple different op modules. This allows for hardware
that only supports some of the operations, not all of them (e.g., a
GPU that only supports single-precision operations).
All the hard-coded back-end MPI_Op functions for (MPI_Op,
MPI_Datatype) tuples still exist, but unlike coll, they're in the
framework base (vs. being in a separate "basic" component) and are
automatically used if no component is found at runtime that provides a
module with the necessary function pointers.
There is an "example" op component that will hopefully be useful to
those writing meaningful op components. It is currently
.ompi_ignore'd so that it doesn't impinge on other developers (it's
somewhat chatty in terms of opal_output() so that you can tell when
its functions have been invoked). See the README file in the example
op component directory. Developers of new op components are
encouraged to look at the following wiki pages:
https://svn.open-mpi.org/trac/ompi/wiki/devel/Autogen
https://svn.open-mpi.org/trac/ompi/wiki/devel/CreateComponent
https://svn.open-mpi.org/trac/ompi/wiki/devel/CreateFramework
= MPI_REDUCE_LOCAL =
Part of the MPI-2.2 proposal listed here:
https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/24
is to add a new function named MPI_REDUCE_LOCAL. It is very easy to
implement, so I added it (also because it makes testing the op
framework pretty easy -- you can do it in serial rather than via
parallel reductions). There's even a man page!
This commit was SVN r20280.
2009-01-14 23:44:31 +00:00
|
|
|
|
|
|
|
return OMPI_SUCCESS;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Query a component, see if it wants to run at all. If it does, save
|
|
|
|
* some information. If it doesn't, close it.
|
|
|
|
*/
|
|
|
|
static int init_query(const mca_base_component_t * c,
|
|
|
|
mca_base_component_priority_list_item_t * entry,
|
|
|
|
bool enable_progress_threads, bool enable_mpi_threads)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
opal_output_verbose(10, ompi_op_base_output,
|
|
|
|
"op:find_available: querying op component %s",
|
|
|
|
c->mca_component_name);
|
|
|
|
|
|
|
|
/* This component has already been successfully opened. So now
|
|
|
|
query it. */
|
|
|
|
|
|
|
|
if (1 == c->mca_type_major_version &&
|
|
|
|
0 == c->mca_type_minor_version &&
|
|
|
|
0 == c->mca_type_release_version) {
|
|
|
|
ret = init_query_1_0_0(c, entry, enable_progress_threads,
|
|
|
|
enable_mpi_threads);
|
|
|
|
} else {
|
|
|
|
/* Unrecognized op API version */
|
|
|
|
|
|
|
|
opal_output_verbose(10, ompi_op_base_output,
|
|
|
|
"op:find_available: unrecognized op API version (%d.%d.%d, ignored)",
|
|
|
|
c->mca_type_major_version,
|
|
|
|
c->mca_type_minor_version,
|
|
|
|
c->mca_type_release_version);
|
|
|
|
return OMPI_ERROR;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Query done -- look at the return value to see what happened */
|
|
|
|
|
|
|
|
if (OMPI_SUCCESS != ret) {
|
|
|
|
opal_output_verbose(10, ompi_op_base_output,
|
|
|
|
"op:find_available: op component %s is not available",
|
|
|
|
c->mca_component_name);
|
|
|
|
if (NULL != c->mca_close_component) {
|
|
|
|
c->mca_close_component();
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
opal_output_verbose(10, ompi_op_base_output,
|
|
|
|
"op:find_available: op component %s is available",
|
|
|
|
c->mca_component_name);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* All done */
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Query a specific component, op v2.0.0
|
|
|
|
*/
|
|
|
|
static int init_query_1_0_0(const mca_base_component_t * component,
|
|
|
|
mca_base_component_priority_list_item_t * entry,
|
|
|
|
bool enable_progress_threads,
|
|
|
|
bool enable_mpi_threads)
|
|
|
|
{
|
|
|
|
ompi_op_base_component_1_0_0_t *op =
|
|
|
|
(ompi_op_base_component_1_0_0_t *) component;
|
|
|
|
|
|
|
|
return op->opc_init_query(enable_progress_threads,
|
|
|
|
enable_mpi_threads);
|
|
|
|
}
|