2004-06-16 19:41:29 +04:00
|
|
|
/*
|
2007-03-17 02:11:45 +03:00
|
|
|
* Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
2005-11-05 22:57:48 +03:00
|
|
|
* University Research and Technology
|
|
|
|
* Corporation. All rights reserved.
|
2006-08-24 20:38:08 +04:00
|
|
|
* Copyright (c) 2004-2006 The University of Tennessee and The University
|
2005-11-05 22:57:48 +03:00
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
* reserved.
|
2008-06-15 17:43:28 +04:00
|
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
2004-11-28 23:09:25 +03:00
|
|
|
* University of Stuttgart. All rights reserved.
|
2005-03-24 15:43:37 +03:00
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
* All rights reserved.
|
2007-07-14 00:49:30 +04:00
|
|
|
* Copyright (c) 2007 Sun Microsystems, Inc. All rights reserved.
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 04:25:21 +04:00
|
|
|
* Copyright (c) 2009 Cisco Systems, Inc. All rights reserved.
|
2010-06-09 20:58:52 +04:00
|
|
|
* Copyright (c) 2010 Los Alamos National Security, LLC.
|
|
|
|
* All rights reserved.
|
2004-11-22 04:38:40 +03:00
|
|
|
* $COPYRIGHT$
|
2008-06-15 17:43:28 +04:00
|
|
|
*
|
2004-11-22 04:38:40 +03:00
|
|
|
* Additional copyrights may follow
|
2008-06-15 17:43:28 +04:00
|
|
|
*
|
2004-06-16 19:41:29 +04:00
|
|
|
* $HEADER$
|
|
|
|
*/
|
|
|
|
/**
|
|
|
|
* @file
|
|
|
|
*/
|
|
|
|
#ifndef MCA_MPOOL_SM_H
|
|
|
|
#define MCA_MPOOL_SM_H
|
|
|
|
|
2009-03-04 18:35:54 +03:00
|
|
|
#include "ompi_config.h"
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 04:25:21 +04:00
|
|
|
|
2005-07-04 03:09:55 +04:00
|
|
|
#include "opal/event/event.h"
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 04:25:21 +04:00
|
|
|
|
2010-06-09 20:58:52 +04:00
|
|
|
#include "ompi/mca/common/sm/common_sm.h"
|
2006-02-12 04:33:29 +03:00
|
|
|
#include "ompi/mca/mpool/mpool.h"
|
|
|
|
#include "ompi/mca/allocator/allocator.h"
|
2006-08-24 20:38:08 +04:00
|
|
|
|
2009-08-20 15:42:18 +04:00
|
|
|
BEGIN_C_DECLS
|
2004-06-16 19:41:29 +04:00
|
|
|
|
|
|
|
struct mca_mpool_sm_component_t {
|
2005-05-31 21:06:55 +04:00
|
|
|
mca_mpool_base_component_t super;
|
|
|
|
/* mca_allocator_base_module_t* sm_allocator; */
|
|
|
|
char* sm_allocator_name;
|
2007-07-14 00:49:30 +04:00
|
|
|
int verbose;
|
2005-05-31 21:06:55 +04:00
|
|
|
/* struct mca_mpool_sm_mmap_t *sm_mmap; */
|
2004-06-16 19:41:29 +04:00
|
|
|
};
|
|
|
|
typedef struct mca_mpool_sm_component_t mca_mpool_sm_component_t;
|
|
|
|
|
2008-06-15 17:43:28 +04:00
|
|
|
typedef struct mca_mpool_base_resources_t {
|
2009-02-20 22:51:57 +03:00
|
|
|
size_t size;
|
2008-06-15 17:43:28 +04:00
|
|
|
int32_t mem_node;
|
|
|
|
} mca_mpool_base_resources_t;
|
|
|
|
|
2006-08-24 20:38:08 +04:00
|
|
|
OMPI_MODULE_DECLSPEC extern mca_mpool_sm_component_t mca_mpool_sm_component;
|
2005-05-31 21:06:55 +04:00
|
|
|
|
2008-06-15 17:43:28 +04:00
|
|
|
typedef struct mca_mpool_sm_module_t {
|
|
|
|
mca_mpool_base_module_t super;
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 04:25:21 +04:00
|
|
|
long sm_size;
|
2008-06-15 17:43:28 +04:00
|
|
|
mca_allocator_base_module_t * sm_allocator;
|
|
|
|
struct mca_mpool_sm_mmap_t *sm_mmap;
|
2010-06-09 20:58:52 +04:00
|
|
|
mca_common_sm_module_t *sm_common_module;
|
2008-06-15 17:43:28 +04:00
|
|
|
int32_t mem_node;
|
|
|
|
} mca_mpool_sm_module_t;
|
2005-05-31 21:06:55 +04:00
|
|
|
|
2008-06-15 17:43:28 +04:00
|
|
|
/*
|
|
|
|
* Initializes the mpool module.
|
|
|
|
*/
|
|
|
|
void mca_mpool_sm_module_init(mca_mpool_sm_module_t* mpool);
|
2005-05-31 21:06:55 +04:00
|
|
|
|
2004-06-18 00:33:52 +04:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Returns base address of shared memory mapping.
|
|
|
|
*/
|
2005-05-31 21:06:55 +04:00
|
|
|
void* mca_mpool_sm_base(mca_mpool_base_module_t*);
|
2004-06-18 00:33:52 +04:00
|
|
|
|
2004-06-16 19:41:29 +04:00
|
|
|
/**
|
2004-06-18 00:33:52 +04:00
|
|
|
* Allocate block of shared memory.
|
2004-06-16 19:41:29 +04:00
|
|
|
*/
|
2008-06-15 17:43:28 +04:00
|
|
|
void* mca_mpool_sm_alloc(
|
|
|
|
mca_mpool_base_module_t* mpool,
|
|
|
|
size_t size,
|
|
|
|
size_t align,
|
2005-09-13 02:28:23 +04:00
|
|
|
uint32_t flags,
|
2005-06-25 01:12:38 +04:00
|
|
|
mca_mpool_base_registration_t** registration);
|
2004-08-02 04:24:22 +04:00
|
|
|
|
2004-06-16 19:41:29 +04:00
|
|
|
/**
|
|
|
|
* realloc function typedef
|
|
|
|
*/
|
2005-06-21 21:10:28 +04:00
|
|
|
void* mca_mpool_sm_realloc(
|
2008-06-15 17:43:28 +04:00
|
|
|
mca_mpool_base_module_t* mpool,
|
|
|
|
void* addr,
|
|
|
|
size_t size,
|
2005-06-25 01:12:38 +04:00
|
|
|
mca_mpool_base_registration_t** registration);
|
2004-08-02 04:24:22 +04:00
|
|
|
|
2004-06-16 19:41:29 +04:00
|
|
|
/**
|
|
|
|
* free function typedef
|
|
|
|
*/
|
2005-06-21 21:10:28 +04:00
|
|
|
void mca_mpool_sm_free(
|
2008-06-15 17:43:28 +04:00
|
|
|
mca_mpool_base_module_t* mpool,
|
|
|
|
void * addr,
|
2005-06-25 01:12:38 +04:00
|
|
|
mca_mpool_base_registration_t* registration);
|
2004-08-02 04:24:22 +04:00
|
|
|
|
2007-03-17 02:11:45 +03:00
|
|
|
/**
|
|
|
|
* Fault Tolerance Event Notification Function
|
|
|
|
* @param state Checkpoint Stae
|
|
|
|
* @return OMPI_SUCCESS or failure status
|
|
|
|
*/
|
|
|
|
int mca_mpool_sm_ft_event(int state);
|
2004-06-16 19:41:29 +04:00
|
|
|
|
2009-08-20 15:42:18 +04:00
|
|
|
END_C_DECLS
|
2004-06-16 19:41:29 +04:00
|
|
|
|
|
|
|
#endif
|