2004-11-22 00:37:56 +00:00
|
|
|
/*
|
2005-11-05 19:57:48 +00:00
|
|
|
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
|
|
|
* University Research and Technology
|
|
|
|
* Corporation. All rights reserved.
|
|
|
|
* Copyright (c) 2004-2005 The University of Tennessee and The University
|
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
* reserved.
|
2010-08-23 16:04:13 +00:00
|
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
2004-11-28 20:09:25 +00:00
|
|
|
* University of Stuttgart. All rights reserved.
|
2005-03-24 12:43:37 +00:00
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
* All rights reserved.
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 00:25:21 +00:00
|
|
|
* Copyright (c) 2009 Cisco Systems, Inc. All rights reserved.
|
2010-08-23 16:04:13 +00:00
|
|
|
* Copyright (c) 2010 Los Alamos National Security, LLC.
|
|
|
|
* All rights reserved.
|
2004-11-22 01:38:40 +00:00
|
|
|
* $COPYRIGHT$
|
2010-08-23 16:04:13 +00:00
|
|
|
*
|
2004-11-22 01:38:40 +00:00
|
|
|
* Additional copyrights may follow
|
2010-08-23 16:04:13 +00:00
|
|
|
*
|
2004-11-22 00:37:56 +00:00
|
|
|
* $HEADER$
|
|
|
|
*/
|
|
|
|
|
2004-08-06 19:35:57 +00:00
|
|
|
#ifndef _COMMON_SM_MMAP_H_
|
|
|
|
#define _COMMON_SM_MMAP_H_
|
2004-06-16 15:41:29 +00:00
|
|
|
|
2005-02-07 22:13:58 +00:00
|
|
|
#include "ompi_config.h"
|
|
|
|
|
2005-07-03 16:06:07 +00:00
|
|
|
#include "opal/class/opal_object.h"
|
2005-07-03 16:22:16 +00:00
|
|
|
#include "opal/class/opal_list.h"
|
2006-02-12 01:33:29 +00:00
|
|
|
#include "opal/sys/atomic.h"
|
2010-08-23 16:04:13 +00:00
|
|
|
#include "ompi/mca/mpool/mpool.h"
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 00:25:21 +00:00
|
|
|
#include "ompi/proc/proc.h"
|
2010-06-09 16:58:52 +00:00
|
|
|
#include "ompi/mca/common/sm/common_sm.h"
|
2005-06-24 21:12:38 +00:00
|
|
|
|
2009-08-20 11:42:18 +00:00
|
|
|
BEGIN_C_DECLS
|
2005-06-21 17:10:28 +00:00
|
|
|
|
|
|
|
struct mca_mpool_base_module_t;
|
|
|
|
|
2010-08-23 16:04:13 +00:00
|
|
|
typedef struct mca_common_sm_module_mmap_t
|
2010-06-09 16:58:52 +00:00
|
|
|
{
|
|
|
|
mca_common_sm_module_t super;
|
|
|
|
} mca_common_sm_module_mmap_t;
|
2004-06-16 15:41:29 +00:00
|
|
|
|
2010-06-09 16:58:52 +00:00
|
|
|
OBJ_CLASS_DECLARATION(mca_common_sm_module_mmap_t);
|
2004-06-16 15:41:29 +00:00
|
|
|
|
2004-08-04 17:22:16 +00:00
|
|
|
/**
|
|
|
|
* This routine is used to set up a shared memory file, backed
|
|
|
|
* by a specified file. It is assumed that the file does not
|
|
|
|
* exist before any of the current set of processes try and open
|
|
|
|
* it.
|
|
|
|
*
|
2010-08-23 16:04:13 +00:00
|
|
|
* @param sorted_procs - array of (ompi_proc_t *)'s to create this shared memory
|
|
|
|
* segment for. this routine, unlike the top-level
|
|
|
|
* mca_common_sm_init routine, assumes that sorted_procs
|
|
|
|
* is in the following state: all the local procs at the
|
|
|
|
* beginning; sorted_procs[0] is the lowest named process.
|
|
|
|
* (IN)
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 00:25:21 +00:00
|
|
|
*
|
2010-08-23 16:04:13 +00:00
|
|
|
* @param num_loc_procs - number of local procs contained within sorted_procs
|
|
|
|
* (IN)
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 00:25:21 +00:00
|
|
|
*
|
2004-08-04 17:22:16 +00:00
|
|
|
* @param size - size of the file, in bytes (IN)
|
|
|
|
*
|
|
|
|
* @param file_name name of file to be opened. (IN)
|
|
|
|
*
|
|
|
|
* @param size_ctl_structure size of the control structure at
|
|
|
|
* the head of the file. The control structure
|
2010-06-09 16:58:52 +00:00
|
|
|
* is assumed to have mca_common_sm_seg_header_t
|
2004-08-04 17:22:16 +00:00
|
|
|
* as its first segment (IN)
|
|
|
|
*
|
|
|
|
* @param data_set_alignment alignment of the data segment. this
|
2010-08-23 16:04:13 +00:00
|
|
|
* follows the control structure. If this
|
|
|
|
* value if 0, then assume that there will
|
|
|
|
* be no data segment following the control
|
2005-08-08 21:38:27 +00:00
|
|
|
* structure. (IN)
|
2004-08-18 15:02:21 +00:00
|
|
|
*
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 00:25:21 +00:00
|
|
|
* @return value pointer to control structure at head of file.
|
2004-08-04 17:22:16 +00:00
|
|
|
*/
|
2010-06-09 16:58:52 +00:00
|
|
|
OMPI_DECLSPEC extern mca_common_sm_module_t *
|
2010-08-23 16:04:13 +00:00
|
|
|
mca_common_sm_mmap_init(ompi_proc_t **sorted_procs,
|
|
|
|
size_t num_loc_procs,
|
|
|
|
size_t size,
|
2010-06-09 16:58:52 +00:00
|
|
|
char *file_name,
|
2010-08-23 16:04:13 +00:00
|
|
|
size_t size_ctl_structure,
|
2010-06-09 16:58:52 +00:00
|
|
|
size_t data_seg_alignment);
|
Fixes trac:1988. The little bug that turned out to be huge. Yoinks.
* Various cosmetic/style updates in the btl sm
* Clean up concept of mpool module (I think that code was written way
back when the concept of "modules" was fuzzy)
* Bring over some old fixes from the /tmp/timattox-sm-coll/ tree to
fix potential segv's when mmap'ed regions were at different
addresses in different processes (thanks Tim!).
* Change sm coll to no longer use mpool as its main source of shmem;
rather, just mmap its own segment (because it's fixed size --
there was nothing to be gained by using mpool; shedding the use of
mpool saved a lot of complexity in the sm coll setup). This
effectively made Tim's fixes moot (because now everything is an
offset into the mmap that is computed locally; there are no global
pointers). :-)
* Slightly updated common/sm to allow making mmap's for a specific
set of procs (vs. ''all'' procs in the process). This potentially
allows for same-host-inter-proc mmaps -- yay!
* Fixed many, many things in the coll sm (particularly in reduce):
* Fixed handling of MPI_IN_PLACE in reduce and allreduce
* Fixed handling of non-contiguous datatypes in reduce
* Changed the order of reductions to go from process (n-1)'s data
to process 0's data, because that's how all other OMPI coll
components work
* Fixed lots of usage of ddt functions
* When using a non-contiguous datatype, if the root process is not
(n-1), now we used a 2nd convertor to copy from shmem to the rbuf
(saves a memory copy vs. what was done before)
* Lots and lots of little cleanups, clarifications, and minor
optimizations (although still more could be done -- e.g., I think
the use of write memory barriers is fairly sub-optimal; they
could be ganged together at the root, for example)
I'm marking this as "fixes trac:1988" and closing the ticket; if something
is still broken, we can re-open the ticket.
This commit was SVN r21967.
The following Trac tickets were found above:
Ticket 1988 --> https://svn.open-mpi.org/trac/ompi/ticket/1988
2009-09-15 00:25:21 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Callback from the sm mpool
|
|
|
|
*/
|
2010-06-09 16:58:52 +00:00
|
|
|
OMPI_DECLSPEC extern void *
|
2010-08-23 16:04:13 +00:00
|
|
|
mca_common_sm_mmap_seg_alloc(struct mca_mpool_base_module_t *mpool,
|
|
|
|
size_t *size,
|
2010-06-09 16:58:52 +00:00
|
|
|
mca_mpool_base_registration_t **registration);
|
2004-08-06 19:35:57 +00:00
|
|
|
|
2005-12-31 15:06:24 +00:00
|
|
|
/**
|
2010-03-09 20:41:25 +00:00
|
|
|
* This function will release all local resources attached to the
|
2005-12-31 15:06:24 +00:00
|
|
|
* mmapped file. We assume that the operating system will destroy the
|
|
|
|
* file when the last process release it.
|
|
|
|
*
|
|
|
|
* @param sm_mmap - the control structure at head of file.
|
|
|
|
*
|
|
|
|
* @returnvalue 0 if everything was OK, otherwise a negative value.
|
|
|
|
*/
|
|
|
|
|
2010-08-23 16:04:13 +00:00
|
|
|
OMPI_DECLSPEC extern int
|
2010-06-09 16:58:52 +00:00
|
|
|
mca_common_sm_mmap_fini(mca_common_sm_module_t *mca_common_sm_module);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* component query routine
|
|
|
|
*/
|
2010-08-23 16:04:13 +00:00
|
|
|
OMPI_DECLSPEC extern int
|
2010-06-09 16:58:52 +00:00
|
|
|
mca_common_sm_mmap_component_query(void);
|
2005-12-31 15:06:24 +00:00
|
|
|
|
2009-08-20 11:42:18 +00:00
|
|
|
END_C_DECLS
|
2004-06-16 15:41:29 +00:00
|
|
|
|
|
|
|
#endif
|
|
|
|
|