Topic/monitoring (#3109)
Add a monitoring PML, OSC and IO. They track all data exchanges between processes,
with capability to include or exclude collective traffic. The monitoring infrastructure is
driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows.
Documentations and examples have been added, as well as a shared library that can be
used with LD_PRELOAD and that allows the monitoring of any application.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* add ability to querry pml monitorinting results with MPI Tools interface
using performance variables "pml_monitoring_messages_count" and
"pml_monitoring_messages_size"
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Fix a convertion problem and add a comment about the lack of component
retain in the new component infrastructure.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Allow the pvar to be written by invoking the associated callback.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Various fixes for the monitoring.
Allocate all counting arrays in a single allocation
Don't delay the initialization (do it at the first add_proc as we
know the number of processes in MPI_COMM_WORLD)
Add a choice: with or without MPI_T (default).
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Cleanup for the monitoring module.
Fixed few bugs, and reshape the operations to prepare for
global or communicator-based monitoring. Start integrating
support for MPI_T as well as MCA monitoring.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Adding documentation about how to use pml_monitoring component.
Document present the use with and without MPI_T.
May not reflect exactly how it works right now, but should reflects
how it should work in the end.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c.
Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Improve monitoring support (including integration with MPI_T)
Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set
Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename
Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output.
Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example
Set filename only if using mpi tools
Adding missing parameters for fprintf in monitoring_flush (for output in std's cases)
Fix expected output/results for example header
Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer
Base whether to output or not on message count, in order to print something if only empty messages are exchanged
Add a new example on how to access performance variables from within the code
Allocate arrays regarding value returned by binding
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add overhead benchmark, with script to use data and create graphs out of the results
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix segfault error at end when not loading pml
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Start create common monitoring module. Factorise version numbering
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix microbenchmarks script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Improve readability of code
NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add osc monitoring component
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add error checking if running out of memory in osc_monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Resolve brutal segfault when double freeing filename
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Moving to ompi/mca/common the proper parts of the monitoring system
Using common functions instead of pml specific one. Removing pml ones.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add calls to record monitored data from osc. Use common function to translate ranks.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix test_overhead benchmark script distribution
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix linking library with mca/common
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add passive operations in monitoring_test
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix from rank calculation. Add more detailed error messages
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix osc_monitoring mget_message_count function call
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add monitoring common output system
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Consistent output file name (with and without MPI_T).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Always output to a file when flushing at pvar_stop(flush).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Update the monitoring documentation.
Complete informations from HowTo. Fix a few mistake and typos.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use the world_rank for printf's.
Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add security check for unique initialization for osc monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean the amout of symbols available outside mca/common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Remove use of __sync_* built-ins. Use opal_atomic_* instead.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Deleting now useless file : moved to common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add histogram ditribution of message sizes
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add coll component for collectives communications monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix log10_2 constant initialization. Fix index calculation for histogram array.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add debug info messages to follow more easily initialization steps.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring.
monitoring_filter only indicates if filtering is activated.
Fix out of range access in histogram.
List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t.
Remove useless dead code.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Don't install the test scripts.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix missing procs in hashtable. Cache coll monitoring data.
* Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer.
* Cache monitoring data relative to collectives operations on creation.
* Remove double caching.
* Use same proc name definition for hash table when inserting and
when retrieving.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use intermediate variable to avoid invalid write while retrieving ranks in hashtable.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add missing release of the last element in flush_all. Add release of the hashtable in finalize.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Set world_rank from hashtable only if found
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use predefined symbol from opal system to print int
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add automated check (with MPI_Tools) of monitoring.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix procs list caching in common_monitoring_coll_data_t
* Fix monitoring_coll_data type definition.
* Use size(COMM_WORLD)-1 to determine max number of digits.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Documentation update.
Update and then move the latex and README documentation to a more logical place
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add the use of a machine file for overhead benchmark
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Check for out-of-bound write in histogram
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix common_monitoring_cache object init for MPI_COMM_WORLD
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add RDMA benchmarks to test_overhead
Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add technical documentation
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Adapt to the new definition of communicators
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Update expected output in test/monitoring/monitoring_test.c
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add dumping histogram in edge case
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add consistency in header inclusion.
Include ompi/mpi/fortran/mpif-h/bindings.h only if needed.
Add sanity check before emptying hashtable.
Fix typos in documentation.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* misc monitoring fixes
* test/monitoring: fix test when weak symbols are not available
* monitoring: fix a typo and add a missing file in Makefile.am
and have monitoring_common.h and monitoring_common_coll.h included in the distro
* test/monitoring: cleanup all tests and make distclean a happy panda
* test/monitoring: use gettimeofday() if clock_gettime() is unavailable
* monitoring: silence misc warnings (#3)
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
* Cleanups.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Changing int64_t to size_t.
Keep the size_t used accross all monitoring components.
Adapt the documentation.
Remove useless MPI_Request and MPI_Status from monitoring_test.c.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add parameter for RMA test case
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean the maximum bound computation for proc list dump.
Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add communicator-specific monitored collective data reset
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add monitoring scripts to the 'make dist'
Also install them in the build and the install directories.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Этот коммит содержится в:
bosilca
GitHub
родитель
b1e639e81e
Коммит
d55b666834
@ -1409,6 +1409,10 @@ AC_CONFIG_FILES([
|
||||
test/util/Makefile
|
||||
])
|
||||
m4_ifdef([project_ompi], [AC_CONFIG_FILES([test/monitoring/Makefile])])
|
||||
m4_ifdef([project_ompi], [
|
||||
m4_ifdef([MCA_BUILD_ompi_pml_monitoring_DSO_TRUE],
|
||||
[AC_CONFIG_LINKS(test/monitoring/profile2mat.pl:test/monitoring/profile2mat.pl
|
||||
test/monitoring/aggregate_profile.pl:test/monitoring/aggregate_profile.pl)])])
|
||||
|
||||
AC_CONFIG_FILES([contrib/dist/mofed/debian/rules],
|
||||
[chmod +x contrib/dist/mofed/debian/rules])
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The University of Tennessee and The University
|
||||
* Copyright (c) 2004-2017 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
||||
@ -46,9 +46,6 @@
|
||||
static int init_query(const mca_base_component_t * ls,
|
||||
bool enable_progress_threads,
|
||||
bool enable_mpi_threads);
|
||||
static int init_query_2_0_0(const mca_base_component_t * ls,
|
||||
bool enable_progress_threads,
|
||||
bool enable_mpi_threads);
|
||||
|
||||
/*
|
||||
* Scan down the list of successfully opened components and query each of
|
||||
@ -105,6 +102,20 @@ int mca_coll_base_find_available(bool enable_progress_threads,
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* Query a specific component, coll v2.0.0
|
||||
*/
|
||||
static inline int
|
||||
init_query_2_0_0(const mca_base_component_t * component,
|
||||
bool enable_progress_threads,
|
||||
bool enable_mpi_threads)
|
||||
{
|
||||
mca_coll_base_component_2_0_0_t *coll =
|
||||
(mca_coll_base_component_2_0_0_t *) component;
|
||||
|
||||
return coll->collm_init_query(enable_progress_threads,
|
||||
enable_mpi_threads);
|
||||
}
|
||||
/*
|
||||
* Query a component, see if it wants to run at all. If it does, save
|
||||
* some information. If it doesn't, close it.
|
||||
@ -138,33 +149,11 @@ static int init_query(const mca_base_component_t * component,
|
||||
}
|
||||
|
||||
/* Query done -- look at the return value to see what happened */
|
||||
|
||||
if (OMPI_SUCCESS != ret) {
|
||||
opal_output_verbose(10, ompi_coll_base_framework.framework_output,
|
||||
"coll:find_available: coll component %s is not available",
|
||||
component->mca_component_name);
|
||||
} else {
|
||||
opal_output_verbose(10, ompi_coll_base_framework.framework_output,
|
||||
"coll:find_available: coll component %s is available",
|
||||
component->mca_component_name);
|
||||
}
|
||||
|
||||
/* All done */
|
||||
opal_output_verbose(10, ompi_coll_base_framework.framework_output,
|
||||
"coll:find_available: coll component %s is %savailable",
|
||||
component->mca_component_name,
|
||||
(OMPI_SUCCESS == ret) ? "": "not ");
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* Query a specific component, coll v2.0.0
|
||||
*/
|
||||
static int init_query_2_0_0(const mca_base_component_t * component,
|
||||
bool enable_progress_threads,
|
||||
bool enable_mpi_threads)
|
||||
{
|
||||
mca_coll_base_component_2_0_0_t *coll =
|
||||
(mca_coll_base_component_2_0_0_t *) component;
|
||||
|
||||
return coll->collm_init_query(enable_progress_threads,
|
||||
enable_mpi_threads);
|
||||
}
|
||||
|
||||
53
ompi/mca/coll/monitoring/Makefile.am
Обычный файл
53
ompi/mca/coll/monitoring/Makefile.am
Обычный файл
@ -0,0 +1,53 @@
|
||||
#
|
||||
# Copyright (c) 2016 Inria. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
#
|
||||
# $HEADER$
|
||||
#
|
||||
|
||||
monitoring_sources = \
|
||||
coll_monitoring.h \
|
||||
coll_monitoring_allgather.c \
|
||||
coll_monitoring_allgatherv.c \
|
||||
coll_monitoring_allreduce.c \
|
||||
coll_monitoring_alltoall.c \
|
||||
coll_monitoring_alltoallv.c \
|
||||
coll_monitoring_alltoallw.c \
|
||||
coll_monitoring_barrier.c \
|
||||
coll_monitoring_bcast.c \
|
||||
coll_monitoring_component.c \
|
||||
coll_monitoring_exscan.c \
|
||||
coll_monitoring_gather.c \
|
||||
coll_monitoring_gatherv.c \
|
||||
coll_monitoring_neighbor_allgather.c \
|
||||
coll_monitoring_neighbor_allgatherv.c \
|
||||
coll_monitoring_neighbor_alltoall.c \
|
||||
coll_monitoring_neighbor_alltoallv.c \
|
||||
coll_monitoring_neighbor_alltoallw.c \
|
||||
coll_monitoring_reduce.c \
|
||||
coll_monitoring_reduce_scatter.c \
|
||||
coll_monitoring_reduce_scatter_block.c \
|
||||
coll_monitoring_scan.c \
|
||||
coll_monitoring_scatter.c \
|
||||
coll_monitoring_scatterv.c
|
||||
|
||||
if MCA_BUILD_ompi_coll_monitoring_DSO
|
||||
component_noinst =
|
||||
component_install = mca_coll_monitoring.la
|
||||
else
|
||||
component_noinst = libmca_coll_monitoring.la
|
||||
component_install =
|
||||
endif
|
||||
|
||||
mcacomponentdir = $(ompilibdir)
|
||||
mcacomponent_LTLIBRARIES = $(component_install)
|
||||
mca_coll_monitoring_la_SOURCES = $(monitoring_sources)
|
||||
mca_coll_monitoring_la_LDFLAGS = -module -avoid-version
|
||||
mca_coll_monitoring_la_LIBADD = \
|
||||
$(OMPI_TOP_BUILDDIR)/ompi/mca/common/monitoring/libmca_common_monitoring.la
|
||||
|
||||
noinst_LTLIBRARIES = $(component_noinst)
|
||||
libmca_coll_monitoring_la_SOURCES = $(monitoring_sources)
|
||||
libmca_coll_monitoring_la_LDFLAGS = -module -avoid-version
|
||||
385
ompi/mca/coll/monitoring/coll_monitoring.h
Обычный файл
385
ompi/mca/coll/monitoring/coll_monitoring.h
Обычный файл
@ -0,0 +1,385 @@
|
||||
/*
|
||||
* Copyright (c) 2016 Inria. All rights reserved.
|
||||
* Copyright (c) 2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_COLL_MONITORING_H
|
||||
#define MCA_COLL_MONITORING_H
|
||||
|
||||
BEGIN_C_DECLS
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/mca/coll/coll.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/mca/common/monitoring/common_monitoring.h>
|
||||
|
||||
struct mca_coll_monitoring_component_t {
|
||||
mca_coll_base_component_t super;
|
||||
int priority;
|
||||
};
|
||||
typedef struct mca_coll_monitoring_component_t mca_coll_monitoring_component_t;
|
||||
|
||||
OMPI_DECLSPEC extern mca_coll_monitoring_component_t mca_coll_monitoring_component;
|
||||
|
||||
struct mca_coll_monitoring_module_t {
|
||||
mca_coll_base_module_t super;
|
||||
mca_coll_base_comm_coll_t real;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
int64_t is_initialized;
|
||||
};
|
||||
typedef struct mca_coll_monitoring_module_t mca_coll_monitoring_module_t;
|
||||
OMPI_DECLSPEC OBJ_CLASS_DECLARATION(mca_coll_monitoring_module_t);
|
||||
|
||||
/*
|
||||
* Coll interface functions
|
||||
*/
|
||||
|
||||
/* Blocking */
|
||||
extern int mca_coll_monitoring_allgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_allgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_allreduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_alltoall(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_alltoallv(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_alltoallw(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_barrier(struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_bcast(void *buff, int count,
|
||||
struct ompi_datatype_t *datatype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_exscan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_gather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount, struct ompi_datatype_t *rdtype,
|
||||
int root, struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_gatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_reduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_reduce_scatter(const void *sbuf, void *rbuf,
|
||||
const int *rcounts,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_reduce_scatter_block(const void *sbuf, void *rbuf,
|
||||
int rcount,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_scan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_scatter(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_scatterv(const void *sbuf, const int *scounts, const int *disps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
/* Nonblocking */
|
||||
extern int mca_coll_monitoring_iallgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_iallgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_iallreduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ialltoall(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ialltoallv(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ialltoallw(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ibarrier(struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ibcast(void *buff, int count,
|
||||
struct ompi_datatype_t *datatype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_iexscan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_igather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount, struct ompi_datatype_t *rdtype,
|
||||
int root, struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_igatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ireduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ireduce_scatter(const void *sbuf, void *rbuf,
|
||||
const int *rcounts,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ireduce_scatter_block(const void *sbuf, void *rbuf,
|
||||
int rcount,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_iscan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_iscatter(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_iscatterv(const void *sbuf, const int *scounts, const int *disps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
/* Neighbor */
|
||||
extern int mca_coll_monitoring_neighbor_allgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype, void *rbuf,
|
||||
int rcount, struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_neighbor_allgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype, void * rbuf,
|
||||
const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_neighbor_alltoall(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_neighbor_alltoallv(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_neighbor_alltoallw(const void *sbuf, const int *scounts,
|
||||
const MPI_Aint *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const MPI_Aint *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ineighbor_allgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype, void *rbuf,
|
||||
int rcount, struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ineighbor_allgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void * rbuf, const int *rcounts,
|
||||
const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ineighbor_alltoall(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype, void *rbuf,
|
||||
int rcount, struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ineighbor_alltoallv(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
extern int mca_coll_monitoring_ineighbor_alltoallw(const void *sbuf, const int *scounts,
|
||||
const MPI_Aint *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const MPI_Aint *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module);
|
||||
|
||||
END_C_DECLS
|
||||
|
||||
#endif /* MCA_COLL_MONITORING_H */
|
||||
71
ompi/mca/coll/monitoring/coll_monitoring_allgather.c
Обычный файл
71
ompi/mca/coll/monitoring/coll_monitoring_allgather.c
Обычный файл
@ -0,0 +1,71 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_allgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( i == my_rank ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, monitoring_module->real.coll_allgather_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_iallgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_iallgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, request, monitoring_module->real.coll_iallgather_module);
|
||||
}
|
||||
71
ompi/mca/coll/monitoring/coll_monitoring_allgatherv.c
Обычный файл
71
ompi/mca/coll/monitoring/coll_monitoring_allgatherv.c
Обычный файл
@ -0,0 +1,71 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_allgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void * rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_allgatherv(sbuf, scount, sdtype, rbuf, rcounts, disps, rdtype, comm, monitoring_module->real.coll_allgatherv_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_iallgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void * rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_iallgatherv(sbuf, scount, sdtype, rbuf, rcounts, disps, rdtype, comm, request, monitoring_module->real.coll_iallgatherv_module);
|
||||
}
|
||||
70
ompi/mca/coll/monitoring/coll_monitoring_allreduce.c
Обычный файл
70
ompi/mca/coll/monitoring/coll_monitoring_allreduce.c
Обычный файл
@ -0,0 +1,70 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_allreduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_allreduce(sbuf, rbuf, count, dtype, op, comm, monitoring_module->real.coll_allreduce_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_iallreduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_iallreduce(sbuf, rbuf, count, dtype, op, comm, request, monitoring_module->real.coll_iallreduce_module);
|
||||
}
|
||||
69
ompi/mca/coll/monitoring/coll_monitoring_alltoall.c
Обычный файл
69
ompi/mca/coll/monitoring/coll_monitoring_alltoall.c
Обычный файл
@ -0,0 +1,69 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_alltoall(const void *sbuf, int scount, struct ompi_datatype_t *sdtype,
|
||||
void* rbuf, int rcount, struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, monitoring_module->real.coll_alltoall_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ialltoall(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_ialltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, request, monitoring_module->real.coll_ialltoall_module);
|
||||
}
|
||||
75
ompi/mca/coll/monitoring/coll_monitoring_alltoallv.c
Обычный файл
75
ompi/mca/coll/monitoring/coll_monitoring_alltoallv.c
Обычный файл
@ -0,0 +1,75 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_alltoallv(const void *sbuf, const int *scounts, const int *sdisps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts, const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
return monitoring_module->real.coll_alltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps, rdtype, comm, monitoring_module->real.coll_alltoallv_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ialltoallv(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
return monitoring_module->real.coll_ialltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps, rdtype, comm, request, monitoring_module->real.coll_ialltoallv_module);
|
||||
}
|
||||
77
ompi/mca/coll/monitoring/coll_monitoring_alltoallw.c
Обычный файл
77
ompi/mca/coll/monitoring/coll_monitoring_alltoallw.c
Обычный файл
@ -0,0 +1,77 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_alltoallw(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
ompi_datatype_type_size(sdtypes[i], &type_size);
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
return monitoring_module->real.coll_alltoallw(sbuf, scounts, sdisps, sdtypes, rbuf, rcounts, rdisps, rdtypes, comm, monitoring_module->real.coll_alltoallw_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ialltoallw(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
ompi_datatype_type_size(sdtypes[i], &type_size);
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
return monitoring_module->real.coll_ialltoallw(sbuf, scounts, sdisps, sdtypes, rbuf, rcounts, rdisps, rdtypes, comm, request, monitoring_module->real.coll_ialltoallw_module);
|
||||
}
|
||||
56
ompi/mca/coll/monitoring/coll_monitoring_barrier.c
Обычный файл
56
ompi/mca/coll/monitoring/coll_monitoring_barrier.c
Обычный файл
@ -0,0 +1,56 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_barrier(struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
int i, rank;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, 0);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(0, monitoring_module->data);
|
||||
return monitoring_module->real.coll_barrier(comm, monitoring_module->real.coll_barrier_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ibarrier(struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
int i, rank;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, 0);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(0, monitoring_module->data);
|
||||
return monitoring_module->real.coll_ibarrier(comm, request, monitoring_module->real.coll_ibarrier_module);
|
||||
}
|
||||
73
ompi/mca/coll/monitoring/coll_monitoring_bcast.c
Обычный файл
73
ompi/mca/coll/monitoring/coll_monitoring_bcast.c
Обычный файл
@ -0,0 +1,73 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_bcast(void *buff, int count,
|
||||
struct ompi_datatype_t *datatype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(datatype, &type_size);
|
||||
data_size = count * type_size;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
mca_common_monitoring_coll_o2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( i == root ) continue; /* No self sending */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_bcast(buff, count, datatype, root, comm, monitoring_module->real.coll_bcast_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ibcast(void *buff, int count,
|
||||
struct ompi_datatype_t *datatype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(datatype, &type_size);
|
||||
data_size = count * type_size;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
mca_common_monitoring_coll_o2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( i == root ) continue; /* No self sending */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_ibcast(buff, count, datatype, root, comm, request, monitoring_module->real.coll_ibcast_module);
|
||||
}
|
||||
255
ompi/mca/coll/monitoring/coll_monitoring_component.c
Обычный файл
255
ompi/mca/coll/monitoring/coll_monitoring_component.c
Обычный файл
@ -0,0 +1,255 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include "coll_monitoring.h"
|
||||
#include <ompi/constants.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/mca/coll/coll.h>
|
||||
#include <opal/mca/base/mca_base_component_repository.h>
|
||||
|
||||
#define MONITORING_SAVE_PREV_COLL_API(__module, __comm, __api) \
|
||||
do { \
|
||||
if( NULL != __comm->c_coll->coll_ ## __api ## _module ) { \
|
||||
__module->real.coll_ ## __api = __comm->c_coll->coll_ ## __api; \
|
||||
__module->real.coll_ ## __api ## _module = __comm->c_coll->coll_ ## __api ## _module; \
|
||||
OBJ_RETAIN(__module->real.coll_ ## __api ## _module); \
|
||||
} else { \
|
||||
/* If no function previously provided, do not monitor */ \
|
||||
__module->super.coll_ ## __api = NULL; \
|
||||
OPAL_MONITORING_PRINT_WARN("COMM \"%s\": No monitoring available for " \
|
||||
"coll_" # __api, __comm->c_name); \
|
||||
} \
|
||||
if( NULL != __comm->c_coll->coll_i ## __api ## _module ) { \
|
||||
__module->real.coll_i ## __api = __comm->c_coll->coll_i ## __api; \
|
||||
__module->real.coll_i ## __api ## _module = __comm->c_coll->coll_i ## __api ## _module; \
|
||||
OBJ_RETAIN(__module->real.coll_i ## __api ## _module); \
|
||||
} else { \
|
||||
/* If no function previously provided, do not monitor */ \
|
||||
__module->super.coll_i ## __api = NULL; \
|
||||
OPAL_MONITORING_PRINT_WARN("COMM \"%s\": No monitoring available for " \
|
||||
"coll_i" # __api, __comm->c_name); \
|
||||
} \
|
||||
} while(0)
|
||||
|
||||
#define MONITORING_RELEASE_PREV_COLL_API(__module, __comm, __api) \
|
||||
do { \
|
||||
if( NULL != __module->real.coll_ ## __api ## _module ) { \
|
||||
if( NULL != __module->real.coll_ ## __api ## _module->coll_module_disable ) { \
|
||||
__module->real.coll_ ## __api ## _module->coll_module_disable(__module->real.coll_ ## __api ## _module, __comm); \
|
||||
} \
|
||||
OBJ_RELEASE(__module->real.coll_ ## __api ## _module); \
|
||||
__module->real.coll_ ## __api = NULL; \
|
||||
__module->real.coll_ ## __api ## _module = NULL; \
|
||||
} \
|
||||
if( NULL != __module->real.coll_i ## __api ## _module ) { \
|
||||
if( NULL != __module->real.coll_i ## __api ## _module->coll_module_disable ) { \
|
||||
__module->real.coll_i ## __api ## _module->coll_module_disable(__module->real.coll_i ## __api ## _module, __comm); \
|
||||
} \
|
||||
OBJ_RELEASE(__module->real.coll_i ## __api ## _module); \
|
||||
__module->real.coll_i ## __api = NULL; \
|
||||
__module->real.coll_i ## __api ## _module = NULL; \
|
||||
} \
|
||||
} while(0)
|
||||
|
||||
#define MONITORING_SET_FULL_PREV_COLL_API(m, c, operation) \
|
||||
do { \
|
||||
operation(m, c, allgather); \
|
||||
operation(m, c, allgatherv); \
|
||||
operation(m, c, allreduce); \
|
||||
operation(m, c, alltoall); \
|
||||
operation(m, c, alltoallv); \
|
||||
operation(m, c, alltoallw); \
|
||||
operation(m, c, barrier); \
|
||||
operation(m, c, bcast); \
|
||||
operation(m, c, exscan); \
|
||||
operation(m, c, gather); \
|
||||
operation(m, c, gatherv); \
|
||||
operation(m, c, reduce); \
|
||||
operation(m, c, reduce_scatter); \
|
||||
operation(m, c, reduce_scatter_block); \
|
||||
operation(m, c, scan); \
|
||||
operation(m, c, scatter); \
|
||||
operation(m, c, scatterv); \
|
||||
operation(m, c, neighbor_allgather); \
|
||||
operation(m, c, neighbor_allgatherv); \
|
||||
operation(m, c, neighbor_alltoall); \
|
||||
operation(m, c, neighbor_alltoallv); \
|
||||
operation(m, c, neighbor_alltoallw); \
|
||||
} while(0)
|
||||
|
||||
#define MONITORING_SAVE_FULL_PREV_COLL_API(m, c) \
|
||||
MONITORING_SET_FULL_PREV_COLL_API((m), (c), MONITORING_SAVE_PREV_COLL_API)
|
||||
|
||||
#define MONITORING_RELEASE_FULL_PREV_COLL_API(m, c) \
|
||||
MONITORING_SET_FULL_PREV_COLL_API((m), (c), MONITORING_RELEASE_PREV_COLL_API)
|
||||
|
||||
static int mca_coll_monitoring_component_open(void)
|
||||
{
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_coll_monitoring_component_close(void)
|
||||
{
|
||||
OPAL_MONITORING_PRINT_INFO("coll_module_close");
|
||||
mca_common_monitoring_finalize();
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_coll_monitoring_component_init(bool enable_progress_threads,
|
||||
bool enable_mpi_threads)
|
||||
{
|
||||
OPAL_MONITORING_PRINT_INFO("coll_module_init");
|
||||
mca_common_monitoring_init();
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_coll_monitoring_component_register(void)
|
||||
{
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int
|
||||
mca_coll_monitoring_module_enable(mca_coll_base_module_t*module, struct ompi_communicator_t*comm)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( 1 == opal_atomic_add_64(&monitoring_module->is_initialized, 1) ) {
|
||||
MONITORING_SAVE_FULL_PREV_COLL_API(monitoring_module, comm);
|
||||
monitoring_module->data = mca_common_monitoring_coll_new(comm);
|
||||
OPAL_MONITORING_PRINT_INFO("coll_module_enabled");
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int
|
||||
mca_coll_monitoring_module_disable(mca_coll_base_module_t*module, struct ompi_communicator_t*comm)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( 0 == opal_atomic_sub_64(&monitoring_module->is_initialized, 1) ) {
|
||||
MONITORING_RELEASE_FULL_PREV_COLL_API(monitoring_module, comm);
|
||||
mca_common_monitoring_coll_release(monitoring_module->data);
|
||||
monitoring_module->data = NULL;
|
||||
OPAL_MONITORING_PRINT_INFO("coll_module_disabled");
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_coll_monitoring_ft_event(int state)
|
||||
{
|
||||
switch(state) {
|
||||
case OPAL_CRS_CHECKPOINT:
|
||||
case OPAL_CRS_CONTINUE:
|
||||
case OPAL_CRS_RESTART:
|
||||
case OPAL_CRS_TERM:
|
||||
default:
|
||||
;
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static mca_coll_base_module_t*
|
||||
mca_coll_monitoring_component_query(struct ompi_communicator_t*comm, int*priority)
|
||||
{
|
||||
OPAL_MONITORING_PRINT_INFO("coll_module_query");
|
||||
mca_coll_monitoring_module_t*monitoring_module = OBJ_NEW(mca_coll_monitoring_module_t);
|
||||
if( NULL == monitoring_module ) return (*priority = -1, NULL);
|
||||
|
||||
/* Initialize module functions */
|
||||
monitoring_module->super.coll_module_enable = mca_coll_monitoring_module_enable;
|
||||
monitoring_module->super.coll_module_disable = mca_coll_monitoring_module_disable;
|
||||
monitoring_module->super.ft_event = mca_coll_monitoring_ft_event;
|
||||
|
||||
/* Initialise module collectives functions */
|
||||
/* Blocking functions */
|
||||
monitoring_module->super.coll_allgather = mca_coll_monitoring_allgather;
|
||||
monitoring_module->super.coll_allgatherv = mca_coll_monitoring_allgatherv;
|
||||
monitoring_module->super.coll_allreduce = mca_coll_monitoring_allreduce;
|
||||
monitoring_module->super.coll_alltoall = mca_coll_monitoring_alltoall;
|
||||
monitoring_module->super.coll_alltoallv = mca_coll_monitoring_alltoallv;
|
||||
monitoring_module->super.coll_alltoallw = mca_coll_monitoring_alltoallw;
|
||||
monitoring_module->super.coll_barrier = mca_coll_monitoring_barrier;
|
||||
monitoring_module->super.coll_bcast = mca_coll_monitoring_bcast;
|
||||
monitoring_module->super.coll_exscan = mca_coll_monitoring_exscan;
|
||||
monitoring_module->super.coll_gather = mca_coll_monitoring_gather;
|
||||
monitoring_module->super.coll_gatherv = mca_coll_monitoring_gatherv;
|
||||
monitoring_module->super.coll_reduce = mca_coll_monitoring_reduce;
|
||||
monitoring_module->super.coll_reduce_scatter = mca_coll_monitoring_reduce_scatter;
|
||||
monitoring_module->super.coll_reduce_scatter_block = mca_coll_monitoring_reduce_scatter_block;
|
||||
monitoring_module->super.coll_scan = mca_coll_monitoring_scan;
|
||||
monitoring_module->super.coll_scatter = mca_coll_monitoring_scatter;
|
||||
monitoring_module->super.coll_scatterv = mca_coll_monitoring_scatterv;
|
||||
|
||||
/* Nonblocking functions */
|
||||
monitoring_module->super.coll_iallgather = mca_coll_monitoring_iallgather;
|
||||
monitoring_module->super.coll_iallgatherv = mca_coll_monitoring_iallgatherv;
|
||||
monitoring_module->super.coll_iallreduce = mca_coll_monitoring_iallreduce;
|
||||
monitoring_module->super.coll_ialltoall = mca_coll_monitoring_ialltoall;
|
||||
monitoring_module->super.coll_ialltoallv = mca_coll_monitoring_ialltoallv;
|
||||
monitoring_module->super.coll_ialltoallw = mca_coll_monitoring_ialltoallw;
|
||||
monitoring_module->super.coll_ibarrier = mca_coll_monitoring_ibarrier;
|
||||
monitoring_module->super.coll_ibcast = mca_coll_monitoring_ibcast;
|
||||
monitoring_module->super.coll_iexscan = mca_coll_monitoring_iexscan;
|
||||
monitoring_module->super.coll_igather = mca_coll_monitoring_igather;
|
||||
monitoring_module->super.coll_igatherv = mca_coll_monitoring_igatherv;
|
||||
monitoring_module->super.coll_ireduce = mca_coll_monitoring_ireduce;
|
||||
monitoring_module->super.coll_ireduce_scatter = mca_coll_monitoring_ireduce_scatter;
|
||||
monitoring_module->super.coll_ireduce_scatter_block = mca_coll_monitoring_ireduce_scatter_block;
|
||||
monitoring_module->super.coll_iscan = mca_coll_monitoring_iscan;
|
||||
monitoring_module->super.coll_iscatter = mca_coll_monitoring_iscatter;
|
||||
monitoring_module->super.coll_iscatterv = mca_coll_monitoring_iscatterv;
|
||||
|
||||
/* Neighborhood functions */
|
||||
monitoring_module->super.coll_neighbor_allgather = mca_coll_monitoring_neighbor_allgather;
|
||||
monitoring_module->super.coll_neighbor_allgatherv = mca_coll_monitoring_neighbor_allgatherv;
|
||||
monitoring_module->super.coll_neighbor_alltoall = mca_coll_monitoring_neighbor_alltoall;
|
||||
monitoring_module->super.coll_neighbor_alltoallv = mca_coll_monitoring_neighbor_alltoallv;
|
||||
monitoring_module->super.coll_neighbor_alltoallw = mca_coll_monitoring_neighbor_alltoallw;
|
||||
monitoring_module->super.coll_ineighbor_allgather = mca_coll_monitoring_ineighbor_allgather;
|
||||
monitoring_module->super.coll_ineighbor_allgatherv = mca_coll_monitoring_ineighbor_allgatherv;
|
||||
monitoring_module->super.coll_ineighbor_alltoall = mca_coll_monitoring_ineighbor_alltoall;
|
||||
monitoring_module->super.coll_ineighbor_alltoallv = mca_coll_monitoring_ineighbor_alltoallv;
|
||||
monitoring_module->super.coll_ineighbor_alltoallw = mca_coll_monitoring_ineighbor_alltoallw;
|
||||
|
||||
/* Initialization flag */
|
||||
monitoring_module->is_initialized = 0;
|
||||
|
||||
*priority = mca_coll_monitoring_component.priority;
|
||||
|
||||
return &(monitoring_module->super);
|
||||
}
|
||||
|
||||
mca_coll_monitoring_component_t mca_coll_monitoring_component = {
|
||||
.super = {
|
||||
/* First, the mca_base_component_t struct containing meta
|
||||
information about the component itself */
|
||||
.collm_version = {
|
||||
MCA_COLL_BASE_VERSION_2_0_0,
|
||||
|
||||
.mca_component_name = "monitoring", /* MCA component name */
|
||||
MCA_MONITORING_MAKE_VERSION,
|
||||
.mca_open_component = mca_coll_monitoring_component_open, /* component open */
|
||||
.mca_close_component = mca_coll_monitoring_component_close, /* component close */
|
||||
.mca_register_component_params = mca_coll_monitoring_component_register
|
||||
},
|
||||
.collm_data = {
|
||||
/* The component is checkpoint ready */
|
||||
MCA_BASE_METADATA_PARAM_CHECKPOINT
|
||||
},
|
||||
|
||||
.collm_init_query = mca_coll_monitoring_component_init,
|
||||
.collm_comm_query = mca_coll_monitoring_component_query
|
||||
},
|
||||
.priority = INT_MAX
|
||||
};
|
||||
|
||||
OBJ_CLASS_INSTANCE(mca_coll_monitoring_module_t,
|
||||
mca_coll_base_module_t,
|
||||
NULL,
|
||||
NULL);
|
||||
|
||||
68
ompi/mca/coll/monitoring/coll_monitoring_exscan.c
Обычный файл
68
ompi/mca/coll/monitoring/coll_monitoring_exscan.c
Обычный файл
@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_exscan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - my_rank), monitoring_module->data);
|
||||
for( i = my_rank + 1; i < comm_size; ++i ) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_exscan(sbuf, rbuf, count, dtype, op, comm, monitoring_module->real.coll_exscan_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_iexscan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - my_rank), monitoring_module->data);
|
||||
for( i = my_rank + 1; i < comm_size; ++i ) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_iexscan(sbuf, rbuf, count, dtype, op, comm, request, monitoring_module->real.coll_iexscan_module);
|
||||
}
|
||||
71
ompi/mca/coll/monitoring/coll_monitoring_gather.c
Обычный файл
71
ompi/mca/coll/monitoring/coll_monitoring_gather.c
Обычный файл
@ -0,0 +1,71 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_gather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount, struct ompi_datatype_t *rdtype,
|
||||
int root, struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(rdtype, &type_size);
|
||||
data_size = rcount * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( root == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2o(data_size * (comm_size - 1), monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_gather(sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm, monitoring_module->real.coll_gather_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_igather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount, struct ompi_datatype_t *rdtype,
|
||||
int root, struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(rdtype, &type_size);
|
||||
data_size = rcount * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( root == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2o(data_size * (comm_size - 1), monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_igather(sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm, request, monitoring_module->real.coll_igather_module);
|
||||
}
|
||||
77
ompi/mca/coll/monitoring/coll_monitoring_gatherv.c
Обычный файл
77
ompi/mca/coll/monitoring/coll_monitoring_gatherv.c
Обычный файл
@ -0,0 +1,77 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_gatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(rdtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( root == i ) continue; /* No communication for self */
|
||||
data_size = rcounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2o(data_size_aggreg, monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_gatherv(sbuf, scount, sdtype, rbuf, rcounts, disps, rdtype, root, comm, monitoring_module->real.coll_gatherv_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_igatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(rdtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( root == i ) continue; /* No communication for self */
|
||||
data_size = rcounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2o(data_size_aggreg, monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_igatherv(sbuf, scount, sdtype, rbuf, rcounts, disps, rdtype, root, comm, request, monitoring_module->real.coll_igatherv_module);
|
||||
}
|
||||
120
ompi/mca/coll/monitoring/coll_monitoring_neighbor_allgather.c
Обычный файл
120
ompi/mca/coll/monitoring/coll_monitoring_neighbor_allgather.c
Обычный файл
@ -0,0 +1,120 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/mca/topo/base/base.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_neighbor_allgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype, void *rbuf,
|
||||
int rcount, struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
|
||||
for( dim = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_neighbor_allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, monitoring_module->real.coll_neighbor_allgather_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ineighbor_allgather(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype, void *rbuf,
|
||||
int rcount, struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
|
||||
for( dim = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_ineighbor_allgather(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, request, monitoring_module->real.coll_ineighbor_allgather_module);
|
||||
}
|
||||
124
ompi/mca/coll/monitoring/coll_monitoring_neighbor_allgatherv.c
Обычный файл
124
ompi/mca/coll/monitoring/coll_monitoring_neighbor_allgatherv.c
Обычный файл
@ -0,0 +1,124 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/mca/topo/base/base.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_neighbor_allgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void * rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_2_2_0_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
|
||||
for( dim = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_neighbor_allgatherv(sbuf, scount, sdtype, rbuf, rcounts, disps, rdtype, comm, monitoring_module->real.coll_neighbor_allgatherv_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ineighbor_allgatherv(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void * rbuf, const int *rcounts, const int *disps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_2_2_0_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
|
||||
for( dim = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_ineighbor_allgatherv(sbuf, scount, sdtype, rbuf, rcounts, disps, rdtype, comm, request, monitoring_module->real.coll_ineighbor_allgatherv_module);
|
||||
}
|
||||
122
ompi/mca/coll/monitoring/coll_monitoring_neighbor_alltoall.c
Обычный файл
122
ompi/mca/coll/monitoring/coll_monitoring_neighbor_alltoall.c
Обычный файл
@ -0,0 +1,122 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/mca/topo/base/base.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_neighbor_alltoall(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void* rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
|
||||
for( dim = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_neighbor_alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, monitoring_module->real.coll_neighbor_alltoall_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ineighbor_alltoall(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
|
||||
for( dim = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_ineighbor_alltoall(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm, request, monitoring_module->real.coll_ineighbor_alltoall_module);
|
||||
}
|
||||
130
ompi/mca/coll/monitoring/coll_monitoring_neighbor_alltoallv.c
Обычный файл
130
ompi/mca/coll/monitoring/coll_monitoring_neighbor_alltoallv.c
Обычный файл
@ -0,0 +1,130 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/mca/topo/base/base.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_neighbor_alltoallv(const void *sbuf, const int *scounts,
|
||||
const int *sdisps, struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts, const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, i, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
|
||||
for( dim = 0, i = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_neighbor_alltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps, rdtype, comm, monitoring_module->real.coll_neighbor_alltoallv_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ineighbor_alltoallv(const void *sbuf, const int *scounts,
|
||||
const int *sdisps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, const int *rcounts,
|
||||
const int *rdisps,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, i, srank, drank, world_rank;
|
||||
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
|
||||
for( dim = 0, i = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_ineighbor_alltoallv(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps, rdtype, comm, request, monitoring_module->real.coll_ineighbor_alltoallv_module);
|
||||
}
|
||||
132
ompi/mca/coll/monitoring/coll_monitoring_neighbor_alltoallw.c
Обычный файл
132
ompi/mca/coll/monitoring/coll_monitoring_neighbor_alltoallw.c
Обычный файл
@ -0,0 +1,132 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/mca/topo/base/base.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_neighbor_alltoallw(const void *sbuf, const int *scounts,
|
||||
const MPI_Aint *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const MPI_Aint *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, i, srank, drank, world_rank;
|
||||
|
||||
for( dim = 0, i = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
ompi_datatype_type_size(sdtypes[i], &type_size);
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
ompi_datatype_type_size(sdtypes[i], &type_size);
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_neighbor_alltoallw(sbuf, scounts, sdisps, sdtypes, rbuf, rcounts, rdisps, rdtypes, comm, monitoring_module->real.coll_neighbor_alltoallw_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ineighbor_alltoallw(const void *sbuf, const int *scounts,
|
||||
const MPI_Aint *sdisps,
|
||||
struct ompi_datatype_t * const *sdtypes,
|
||||
void *rbuf, const int *rcounts,
|
||||
const MPI_Aint *rdisps,
|
||||
struct ompi_datatype_t * const *rdtypes,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const mca_topo_base_comm_cart_t *cart = comm->c_topo->mtc.cart;
|
||||
int dim, i, srank, drank, world_rank;
|
||||
|
||||
for( dim = 0, i = 0; dim < cart->ndims; ++dim ) {
|
||||
srank = MPI_PROC_NULL, drank = MPI_PROC_NULL;
|
||||
|
||||
if (cart->dims[dim] > 1) {
|
||||
mca_topo_base_cart_shift (comm, dim, 1, &srank, &drank);
|
||||
} else if (1 == cart->dims[dim] && cart->periods[dim]) {
|
||||
/* Don't record exchanges with self */
|
||||
continue;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != srank) {
|
||||
ompi_datatype_type_size(sdtypes[i], &type_size);
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(srank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
|
||||
if (MPI_PROC_NULL != drank) {
|
||||
ompi_datatype_type_size(sdtypes[i], &type_size);
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(drank, comm, &world_rank) ) {
|
||||
mca_common_monitoring_record_coll(world_rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
++i;
|
||||
}
|
||||
}
|
||||
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
|
||||
return monitoring_module->real.coll_ineighbor_alltoallw(sbuf, scounts, sdisps, sdtypes, rbuf, rcounts, rdisps, rdtypes, comm, request, monitoring_module->real.coll_ineighbor_alltoallw_module);
|
||||
}
|
||||
74
ompi/mca/coll/monitoring/coll_monitoring_reduce.c
Обычный файл
74
ompi/mca/coll/monitoring/coll_monitoring_reduce.c
Обычный файл
@ -0,0 +1,74 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_reduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( root == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2o(data_size * (comm_size - 1), monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_reduce(sbuf, rbuf, count, dtype, op, root, comm, monitoring_module->real.coll_reduce_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ireduce(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
if( root == ompi_comm_rank(comm) ) {
|
||||
int i, rank;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( root == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2o(data_size * (comm_size - 1), monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_ireduce(sbuf, rbuf, count, dtype, op, root, comm, request, monitoring_module->real.coll_ireduce_module);
|
||||
}
|
||||
74
ompi/mca/coll/monitoring/coll_monitoring_reduce_scatter.c
Обычный файл
74
ompi/mca/coll/monitoring/coll_monitoring_reduce_scatter.c
Обычный файл
@ -0,0 +1,74 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_reduce_scatter(const void *sbuf, void *rbuf,
|
||||
const int *rcounts,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
data_size = rcounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
return monitoring_module->real.coll_reduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm, monitoring_module->real.coll_reduce_scatter_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ireduce_scatter(const void *sbuf, void *rbuf,
|
||||
const int *rcounts,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
data_size = rcounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size_aggreg, monitoring_module->data);
|
||||
return monitoring_module->real.coll_ireduce_scatter(sbuf, rbuf, rcounts, dtype, op, comm, request, monitoring_module->real.coll_ireduce_scatter_module);
|
||||
}
|
||||
72
ompi/mca/coll/monitoring/coll_monitoring_reduce_scatter_block.c
Обычный файл
72
ompi/mca/coll/monitoring/coll_monitoring_reduce_scatter_block.c
Обычный файл
@ -0,0 +1,72 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_reduce_scatter_block(const void *sbuf, void *rbuf,
|
||||
int rcount,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = rcount * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
return monitoring_module->real.coll_reduce_scatter_block(sbuf, rbuf, rcount, dtype, op, comm, monitoring_module->real.coll_reduce_scatter_block_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_ireduce_scatter_block(const void *sbuf, void *rbuf,
|
||||
int rcount,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = rcount * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
return monitoring_module->real.coll_ireduce_scatter_block(sbuf, rbuf, rcount, dtype, op, comm, request, monitoring_module->real.coll_ireduce_scatter_block_module);
|
||||
}
|
||||
68
ompi/mca/coll/monitoring/coll_monitoring_scan.c
Обычный файл
68
ompi/mca/coll/monitoring/coll_monitoring_scan.c
Обычный файл
@ -0,0 +1,68 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_scan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - my_rank), monitoring_module->data);
|
||||
for( i = my_rank + 1; i < comm_size; ++i ) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_scan(sbuf, rbuf, count, dtype, op, comm, monitoring_module->real.coll_scan_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_iscan(const void *sbuf, void *rbuf, int count,
|
||||
struct ompi_datatype_t *dtype,
|
||||
struct ompi_op_t *op,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(dtype, &type_size);
|
||||
data_size = count * type_size;
|
||||
mca_common_monitoring_coll_a2a(data_size * (comm_size - my_rank), monitoring_module->data);
|
||||
for( i = my_rank + 1; i < comm_size; ++i ) {
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
return monitoring_module->real.coll_iscan(sbuf, rbuf, count, dtype, op, comm, request, monitoring_module->real.coll_iscan_module);
|
||||
}
|
||||
78
ompi/mca/coll/monitoring/coll_monitoring_scatter.c
Обычный файл
78
ompi/mca/coll/monitoring/coll_monitoring_scatter.c
Обычный файл
@ -0,0 +1,78 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_scatter(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
if( root == my_rank ) {
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_o2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_scatter(sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm, monitoring_module->real.coll_scatter_module);
|
||||
}
|
||||
|
||||
|
||||
int mca_coll_monitoring_iscatter(const void *sbuf, int scount,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount,
|
||||
struct ompi_datatype_t *rdtype,
|
||||
int root,
|
||||
struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
if( root == my_rank ) {
|
||||
size_t type_size, data_size;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
data_size = scount * type_size;
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
if( my_rank == i ) continue; /* No communication for self */
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_o2a(data_size * (comm_size - 1), monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_iscatter(sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm, request, monitoring_module->real.coll_iscatter_module);
|
||||
}
|
||||
73
ompi/mca/coll/monitoring/coll_monitoring_scatterv.c
Обычный файл
73
ompi/mca/coll/monitoring/coll_monitoring_scatterv.c
Обычный файл
@ -0,0 +1,73 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include "coll_monitoring.h"
|
||||
|
||||
int mca_coll_monitoring_scatterv(const void *sbuf, const int *scounts, const int *disps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void* rbuf, int rcount, struct ompi_datatype_t *rdtype,
|
||||
int root, struct ompi_communicator_t *comm,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
if( root == my_rank ) {
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_o2a(data_size_aggreg, monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_scatterv(sbuf, scounts, disps, sdtype, rbuf, rcount, rdtype, root, comm, monitoring_module->real.coll_scatterv_module);
|
||||
}
|
||||
|
||||
int mca_coll_monitoring_iscatterv(const void *sbuf, const int *scounts, const int *disps,
|
||||
struct ompi_datatype_t *sdtype,
|
||||
void *rbuf, int rcount, struct ompi_datatype_t *rdtype,
|
||||
int root, struct ompi_communicator_t *comm,
|
||||
ompi_request_t ** request,
|
||||
mca_coll_base_module_t *module)
|
||||
{
|
||||
mca_coll_monitoring_module_t*monitoring_module = (mca_coll_monitoring_module_t*) module;
|
||||
const int my_rank = ompi_comm_rank(comm);
|
||||
if( root == my_rank ) {
|
||||
size_t type_size, data_size, data_size_aggreg = 0;
|
||||
const int comm_size = ompi_comm_size(comm);
|
||||
int i, rank;
|
||||
ompi_datatype_type_size(sdtype, &type_size);
|
||||
for( i = 0; i < comm_size; ++i ) {
|
||||
data_size = scounts[i] * type_size;
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if( OPAL_SUCCESS == mca_common_monitoring_get_world_rank(i, comm, &rank) ) {
|
||||
mca_common_monitoring_record_coll(rank, data_size);
|
||||
data_size_aggreg += data_size;
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_o2a(data_size_aggreg, monitoring_module->data);
|
||||
}
|
||||
return monitoring_module->real.coll_iscatterv(sbuf, scounts, disps, sdtype, rbuf, rcount, rdtype, root, comm, request, monitoring_module->real.coll_iscatterv_module);
|
||||
}
|
||||
1298
ompi/mca/common/monitoring/HowTo_pml_monitoring.tex
Обычный файл
1298
ompi/mca/common/monitoring/HowTo_pml_monitoring.tex
Обычный файл
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
50
ompi/mca/common/monitoring/Makefile.am
Обычный файл
50
ompi/mca/common/monitoring/Makefile.am
Обычный файл
@ -0,0 +1,50 @@
|
||||
#
|
||||
# Copyright (c) 2016 Inria. All rights reserved.
|
||||
# Copyright (c) 2017 Research Organization for Information Science
|
||||
# and Technology (RIST). All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
#
|
||||
# $HEADER$
|
||||
#
|
||||
|
||||
sources = common_monitoring.c common_monitoring_coll.c
|
||||
headers = common_monitoring.h common_monitoring_coll.h
|
||||
|
||||
lib_LTLIBRARIES =
|
||||
noinst_LTLIBRARIES =
|
||||
component_install = libmca_common_monitoring.la
|
||||
component_noinst = libmca_common_monitoring_noinst.la
|
||||
|
||||
if MCA_BUILD_ompi_common_monitoring_DSO
|
||||
lib_LTLIBRARIES += $(component_install)
|
||||
else
|
||||
noinst_LTLIBRARIES += $(component_noinst)
|
||||
endif
|
||||
|
||||
libmca_common_monitoring_la_SOURCES = $(headers) $(sources)
|
||||
libmca_common_monitoring_la_CPPFLAGS = $(common_monitoring_CPPFLAGS)
|
||||
libmca_common_monitoring_la_LDFLAGS = \
|
||||
$(common_monitoring_LDFLAGS)
|
||||
libmca_common_monitoring_la_LIBADD = $(common_monitoring_LIBS)
|
||||
libmca_common_monitoring_noinst_la_SOURCES = $(headers) $(sources)
|
||||
|
||||
# These two rules will sym link the "noinst" libtool library filename
|
||||
# to the installable libtool library filename in the case where we are
|
||||
# compiling this component statically (case 2), described above).
|
||||
V=0
|
||||
OMPI_V_LN_SCOMP = $(ompi__v_LN_SCOMP_$V)
|
||||
ompi__v_LN_SCOMP_ = $(ompi__v_LN_SCOMP_$AM_DEFAULT_VERBOSITY)
|
||||
ompi__v_LN_SCOMP_0 = @echo " LN_S " `basename $(component_install)`;
|
||||
|
||||
all-local:
|
||||
$(OMPI_V_LN_SCOMP) if test -z "$(lib_LTLIBRARIES)"; then \
|
||||
rm -f "$(component_install)"; \
|
||||
$(LN_S) "$(component_noinst)" "$(component_install)"; \
|
||||
fi
|
||||
|
||||
clean-local:
|
||||
if test -z "$(lib_LTLIBRARIES)"; then \
|
||||
rm -f "$(component_install)"; \
|
||||
fi
|
||||
795
ompi/mca/common/monitoring/common_monitoring.c
Обычный файл
795
ompi/mca/common/monitoring/common_monitoring.c
Обычный файл
@ -0,0 +1,795 @@
|
||||
/*
|
||||
* Copyright (c) 2013-2017 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2015 Bull SAS. All rights reserved.
|
||||
* Copyright (c) 2016-2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include "common_monitoring.h"
|
||||
#include "common_monitoring_coll.h"
|
||||
#include <ompi/constants.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <opal/mca/base/mca_base_component_repository.h>
|
||||
#include <opal/class/opal_hash_table.h>
|
||||
#include <opal/util/output.h>
|
||||
#include <math.h>
|
||||
|
||||
#if SIZEOF_LONG_LONG == SIZEOF_SIZE_T
|
||||
#define MCA_MONITORING_VAR_TYPE MCA_BASE_VAR_TYPE_UNSIGNED_LONG_LONG
|
||||
#elif SIZEOF_LONG == SIZEOF_SIZE_T
|
||||
#define MCA_MONITORING_VAR_TYPE MCA_BASE_VAR_TYPE_UNSIGNED_LONG
|
||||
#endif
|
||||
|
||||
/*** Monitoring specific variables ***/
|
||||
/* Keep tracks of how many components are currently using the common part */
|
||||
static int32_t mca_common_monitoring_hold = 0;
|
||||
/* Output parameters */
|
||||
int mca_common_monitoring_output_stream_id = -1;
|
||||
static opal_output_stream_t mca_common_monitoring_output_stream_obj = {
|
||||
.lds_verbose_level = 0,
|
||||
.lds_want_syslog = false,
|
||||
.lds_prefix = NULL,
|
||||
.lds_suffix = NULL,
|
||||
.lds_is_debugging = true,
|
||||
.lds_want_stdout = false,
|
||||
.lds_want_stderr = true,
|
||||
.lds_want_file = false,
|
||||
.lds_want_file_append = false,
|
||||
.lds_file_suffix = NULL
|
||||
};
|
||||
|
||||
/*** MCA params to mark the monitoring as enabled. ***/
|
||||
/* This signals that the monitoring will highjack the PML, OSC and COLL */
|
||||
int mca_common_monitoring_enabled = 0;
|
||||
int mca_common_monitoring_current_state = 0;
|
||||
/* Signals there will be an output of the monitored data at component close */
|
||||
static int mca_common_monitoring_output_enabled = 0;
|
||||
/* File where to output the monitored data */
|
||||
static char* mca_common_monitoring_initial_filename = "";
|
||||
static char* mca_common_monitoring_current_filename = NULL;
|
||||
|
||||
/* array for stroring monitoring data*/
|
||||
static size_t* pml_data = NULL;
|
||||
static size_t* pml_count = NULL;
|
||||
static size_t* filtered_pml_data = NULL;
|
||||
static size_t* filtered_pml_count = NULL;
|
||||
static size_t* osc_data_s = NULL;
|
||||
static size_t* osc_count_s = NULL;
|
||||
static size_t* osc_data_r = NULL;
|
||||
static size_t* osc_count_r = NULL;
|
||||
static size_t* coll_data = NULL;
|
||||
static size_t* coll_count = NULL;
|
||||
|
||||
static size_t* size_histogram = NULL;
|
||||
static const int max_size_histogram = 66;
|
||||
static double log10_2 = 0.;
|
||||
|
||||
static int rank_world = -1;
|
||||
static int nprocs_world = 0;
|
||||
|
||||
opal_hash_table_t *common_monitoring_translation_ht = NULL;
|
||||
|
||||
/* Reset all the monitoring arrays */
|
||||
static void mca_common_monitoring_reset ( void );
|
||||
|
||||
/* Flushes the monitored data and reset the values */
|
||||
static int mca_common_monitoring_flush (int fd, char* filename);
|
||||
|
||||
/* Retreive the PML recorded count of messages sent */
|
||||
static int mca_common_monitoring_get_pml_count (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Retreive the PML recorded amount of data sent */
|
||||
static int mca_common_monitoring_get_pml_size (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Retreive the OSC recorded count of messages sent */
|
||||
static int mca_common_monitoring_get_osc_sent_count (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Retreive the OSC recorded amount of data sent */
|
||||
static int mca_common_monitoring_get_osc_sent_size (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Retreive the OSC recorded count of messages received */
|
||||
static int mca_common_monitoring_get_osc_recv_count (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Retreive the OSC recorded amount of data received */
|
||||
static int mca_common_monitoring_get_osc_recv_size (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Retreive the COLL recorded count of messages sent */
|
||||
static int mca_common_monitoring_get_coll_count (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Retreive the COLL recorded amount of data sent */
|
||||
static int mca_common_monitoring_get_coll_size (const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj_handle);
|
||||
|
||||
/* Set the filename where to output the monitored data */
|
||||
static int mca_common_monitoring_set_flush(struct mca_base_pvar_t *pvar,
|
||||
const void *value, void *obj);
|
||||
|
||||
/* Does nothing, as the pml_monitoring_flush pvar has no point to be read */
|
||||
static int mca_common_monitoring_get_flush(const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj);
|
||||
|
||||
/* pml_monitoring_count, pml_monitoring_size,
|
||||
osc_monitoring_sent_count, osc_monitoring sent_size,
|
||||
osc_monitoring_recv_size and osc_monitoring_recv_count pvar notify
|
||||
function */
|
||||
static int mca_common_monitoring_comm_size_notify(mca_base_pvar_t *pvar,
|
||||
mca_base_pvar_event_t event,
|
||||
void *obj_handle, int *count);
|
||||
|
||||
/* pml_monitoring_flush pvar notify function */
|
||||
static int mca_common_monitoring_notify_flush(struct mca_base_pvar_t *pvar,
|
||||
mca_base_pvar_event_t event,
|
||||
void *obj, int *count);
|
||||
|
||||
static int mca_common_monitoring_set_flush(struct mca_base_pvar_t *pvar,
|
||||
const void *value, void *obj)
|
||||
{
|
||||
if( NULL != mca_common_monitoring_current_filename ) {
|
||||
free(mca_common_monitoring_current_filename);
|
||||
}
|
||||
if( NULL == *(char**)value || 0 == strlen((char*)value) ) { /* No more output */
|
||||
mca_common_monitoring_current_filename = NULL;
|
||||
} else {
|
||||
mca_common_monitoring_current_filename = strdup((char*)value);
|
||||
if( NULL == mca_common_monitoring_current_filename )
|
||||
return OMPI_ERROR;
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_flush(const struct mca_base_pvar_t *pvar,
|
||||
void *value, void *obj)
|
||||
{
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_notify_flush(struct mca_base_pvar_t *pvar,
|
||||
mca_base_pvar_event_t event,
|
||||
void *obj, int *count)
|
||||
{
|
||||
switch (event) {
|
||||
case MCA_BASE_PVAR_HANDLE_BIND:
|
||||
mca_common_monitoring_reset();
|
||||
*count = (NULL == mca_common_monitoring_current_filename
|
||||
? 0 : strlen(mca_common_monitoring_current_filename));
|
||||
case MCA_BASE_PVAR_HANDLE_UNBIND:
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_START:
|
||||
mca_common_monitoring_current_state = mca_common_monitoring_enabled;
|
||||
mca_common_monitoring_output_enabled = 0; /* we can't control the monitoring via MPIT and
|
||||
* expect accurate answer upon MPI_Finalize. */
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_STOP:
|
||||
return mca_common_monitoring_flush(3, mca_common_monitoring_current_filename);
|
||||
}
|
||||
return OMPI_ERROR;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_comm_size_notify(mca_base_pvar_t *pvar,
|
||||
mca_base_pvar_event_t event,
|
||||
void *obj_handle,
|
||||
int *count)
|
||||
{
|
||||
switch (event) {
|
||||
case MCA_BASE_PVAR_HANDLE_BIND:
|
||||
/* Return the size of the communicator as the number of values */
|
||||
*count = ompi_comm_size ((ompi_communicator_t *) obj_handle);
|
||||
case MCA_BASE_PVAR_HANDLE_UNBIND:
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_START:
|
||||
mca_common_monitoring_current_state = mca_common_monitoring_enabled;
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_STOP:
|
||||
mca_common_monitoring_current_state = 0;
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
return OMPI_ERROR;
|
||||
}
|
||||
|
||||
void mca_common_monitoring_init( void )
|
||||
{
|
||||
if( mca_common_monitoring_enabled &&
|
||||
1 < opal_atomic_add_32(&mca_common_monitoring_hold, 1) ) return; /* Already initialized */
|
||||
|
||||
char hostname[OPAL_MAXHOSTNAMELEN] = "NA";
|
||||
/* Initialize constant */
|
||||
log10_2 = log10(2.);
|
||||
/* Open the opal_output stream */
|
||||
gethostname(hostname, sizeof(hostname));
|
||||
asprintf(&mca_common_monitoring_output_stream_obj.lds_prefix,
|
||||
"[%s:%06d] monitoring: ", hostname, getpid());
|
||||
mca_common_monitoring_output_stream_id =
|
||||
opal_output_open(&mca_common_monitoring_output_stream_obj);
|
||||
/* Initialize proc translation hashtable */
|
||||
common_monitoring_translation_ht = OBJ_NEW(opal_hash_table_t);
|
||||
opal_hash_table_init(common_monitoring_translation_ht, 2048);
|
||||
}
|
||||
|
||||
void mca_common_monitoring_finalize( void )
|
||||
{
|
||||
if( ! mca_common_monitoring_enabled || /* Don't release if not last */
|
||||
0 < opal_atomic_sub_32(&mca_common_monitoring_hold, 1) ) return;
|
||||
|
||||
OPAL_MONITORING_PRINT_INFO("common_component_finish");
|
||||
/* Dump monitoring informations */
|
||||
mca_common_monitoring_flush(mca_common_monitoring_output_enabled,
|
||||
mca_common_monitoring_current_filename);
|
||||
/* Disable all monitoring */
|
||||
mca_common_monitoring_enabled = 0;
|
||||
/* Close the opal_output stream */
|
||||
opal_output_close(mca_common_monitoring_output_stream_id);
|
||||
free(mca_common_monitoring_output_stream_obj.lds_prefix);
|
||||
/* Free internal data structure */
|
||||
free(pml_data); /* a single allocation */
|
||||
opal_hash_table_remove_all( common_monitoring_translation_ht );
|
||||
OBJ_RELEASE(common_monitoring_translation_ht);
|
||||
mca_common_monitoring_coll_finalize();
|
||||
if( NULL != mca_common_monitoring_current_filename ) {
|
||||
free(mca_common_monitoring_current_filename);
|
||||
mca_common_monitoring_current_filename = NULL;
|
||||
}
|
||||
}
|
||||
|
||||
void mca_common_monitoring_register(void*pml_monitoring_component)
|
||||
{
|
||||
/* Because we are playing tricks with the component close, we should not
|
||||
* use mca_base_component_var_register but instead stay with the basic
|
||||
* version mca_base_var_register.
|
||||
*/
|
||||
(void)mca_base_var_register("ompi", "pml", "monitoring", "enable",
|
||||
"Enable the monitoring at the PML level. A value of 0 "
|
||||
"will disable the monitoring (default). A value of 1 will "
|
||||
"aggregate all monitoring information (point-to-point and "
|
||||
"collective). Any other value will enable filtered monitoring",
|
||||
MCA_BASE_VAR_TYPE_INT, NULL, MPI_T_BIND_NO_OBJECT,
|
||||
MCA_BASE_VAR_FLAG_DWG, OPAL_INFO_LVL_4,
|
||||
MCA_BASE_VAR_SCOPE_READONLY,
|
||||
&mca_common_monitoring_enabled);
|
||||
|
||||
mca_common_monitoring_current_state = mca_common_monitoring_enabled;
|
||||
|
||||
(void)mca_base_var_register("ompi", "pml", "monitoring", "enable_output",
|
||||
"Enable the PML monitoring textual output at MPI_Finalize "
|
||||
"(it will be automatically turned off when MPIT is used to "
|
||||
"monitor communications). This value should be different "
|
||||
"than 0 in order for the output to be enabled (default disable)",
|
||||
MCA_BASE_VAR_TYPE_INT, NULL, MPI_T_BIND_NO_OBJECT,
|
||||
MCA_BASE_VAR_FLAG_DWG, OPAL_INFO_LVL_9,
|
||||
MCA_BASE_VAR_SCOPE_READONLY,
|
||||
&mca_common_monitoring_output_enabled);
|
||||
|
||||
(void)mca_base_var_register("ompi", "pml", "monitoring", "filename",
|
||||
/*&mca_common_monitoring_component.pmlm_version, "filename",*/
|
||||
"The name of the file where the monitoring information "
|
||||
"should be saved (the filename will be extended with the "
|
||||
"process rank and the \".prof\" extension). If this field "
|
||||
"is NULL the monitoring will not be saved.",
|
||||
MCA_BASE_VAR_TYPE_STRING, NULL, MPI_T_BIND_NO_OBJECT,
|
||||
MCA_BASE_VAR_FLAG_DWG, OPAL_INFO_LVL_9,
|
||||
MCA_BASE_VAR_SCOPE_READONLY,
|
||||
&mca_common_monitoring_initial_filename);
|
||||
|
||||
/* Now that the MCA variables are automatically unregistered when
|
||||
* their component close, we need to keep a safe copy of the
|
||||
* filename.
|
||||
* Keep the copy completely separated in order to let the initial
|
||||
* filename to be handled by the framework. It's easier to deal
|
||||
* with the string lifetime.
|
||||
*/
|
||||
if( NULL != mca_common_monitoring_initial_filename )
|
||||
mca_common_monitoring_current_filename = strdup(mca_common_monitoring_initial_filename);
|
||||
|
||||
/* Register PVARs */
|
||||
|
||||
/* PML PVARs */
|
||||
(void)mca_base_pvar_register("ompi", "pml", "monitoring", "flush", "Flush the monitoring "
|
||||
"information in the provided file. The filename is append with "
|
||||
"the .%d.prof suffix, where %d is replaced with the processus "
|
||||
"rank in MPI_COMM_WORLD.",
|
||||
OPAL_INFO_LVL_1, MCA_BASE_PVAR_CLASS_GENERIC,
|
||||
MCA_BASE_VAR_TYPE_STRING, NULL, MPI_T_BIND_NO_OBJECT, 0,
|
||||
mca_common_monitoring_get_flush, mca_common_monitoring_set_flush,
|
||||
mca_common_monitoring_notify_flush, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "pml", "monitoring", "messages_count", "Number of "
|
||||
"messages sent to each peer through the PML framework.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_pml_count, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "pml", "monitoring", "messages_size", "Size of messages "
|
||||
"sent to each peer in a communicator through the PML framework.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_pml_size, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
/* OSC PVARs */
|
||||
(void)mca_base_pvar_register("ompi", "osc", "monitoring", "messages_sent_count", "Number of "
|
||||
"messages sent through the OSC framework with each peer.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_osc_sent_count, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "osc", "monitoring", "messages_sent_size", "Size of "
|
||||
"messages sent through the OSC framework with each peer.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_osc_sent_size, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "osc", "monitoring", "messages_recv_count", "Number of "
|
||||
"messages received through the OSC framework with each peer.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_osc_recv_count, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "osc", "monitoring", "messages_recv_size", "Size of "
|
||||
"messages received through the OSC framework with each peer.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_osc_recv_size, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
/* COLL PVARs */
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "messages_count", "Number of "
|
||||
"messages exchanged through the COLL framework with each peer.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_coll_count, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "messages_size", "Size of "
|
||||
"messages exchanged through the COLL framework with each peer.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_get_coll_size, NULL,
|
||||
mca_common_monitoring_comm_size_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "o2a_count", "Number of messages "
|
||||
"exchanged as one-to-all operations in a communicator.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_COUNTER,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_coll_get_o2a_count, NULL,
|
||||
mca_common_monitoring_coll_messages_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "o2a_size", "Size of messages "
|
||||
"exchanged as one-to-all operations in a communicator.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_AGGREGATE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_coll_get_o2a_size, NULL,
|
||||
mca_common_monitoring_coll_messages_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "a2o_count", "Number of messages "
|
||||
"exchanged as all-to-one operations in a communicator.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_COUNTER,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_coll_get_a2o_count, NULL,
|
||||
mca_common_monitoring_coll_messages_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "a2o_size", "Size of messages "
|
||||
"exchanged as all-to-one operations in a communicator.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_AGGREGATE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_coll_get_a2o_size, NULL,
|
||||
mca_common_monitoring_coll_messages_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "a2a_count", "Number of messages "
|
||||
"exchanged as all-to-all operations in a communicator.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_COUNTER,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_coll_get_a2a_count, NULL,
|
||||
mca_common_monitoring_coll_messages_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "coll", "monitoring", "a2a_size", "Size of messages "
|
||||
"exchanged as all-to-all operations in a communicator.",
|
||||
OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_AGGREGATE,
|
||||
MCA_MONITORING_VAR_TYPE, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_common_monitoring_coll_get_a2a_size, NULL,
|
||||
mca_common_monitoring_coll_messages_notify, NULL);
|
||||
}
|
||||
|
||||
/**
|
||||
* This PML monitors only the processes in the MPI_COMM_WORLD. As OMPI is now lazily
|
||||
* adding peers on the first call to add_procs we need to check how many processes
|
||||
* are in the MPI_COMM_WORLD to create the storage with the right size.
|
||||
*/
|
||||
int mca_common_monitoring_add_procs(struct ompi_proc_t **procs,
|
||||
size_t nprocs)
|
||||
{
|
||||
opal_process_name_t tmp, wp_name;
|
||||
size_t i;
|
||||
int peer_rank;
|
||||
uint64_t key;
|
||||
if( 0 > rank_world )
|
||||
rank_world = ompi_comm_rank((ompi_communicator_t*)&ompi_mpi_comm_world);
|
||||
if( !nprocs_world )
|
||||
nprocs_world = ompi_comm_size((ompi_communicator_t*)&ompi_mpi_comm_world);
|
||||
|
||||
if( NULL == pml_data ) {
|
||||
int array_size = (10 + max_size_histogram) * nprocs_world;
|
||||
pml_data = (size_t*)calloc(array_size, sizeof(size_t));
|
||||
pml_count = pml_data + nprocs_world;
|
||||
filtered_pml_data = pml_count + nprocs_world;
|
||||
filtered_pml_count = filtered_pml_data + nprocs_world;
|
||||
osc_data_s = filtered_pml_count + nprocs_world;
|
||||
osc_count_s = osc_data_s + nprocs_world;
|
||||
osc_data_r = osc_count_s + nprocs_world;
|
||||
osc_count_r = osc_data_r + nprocs_world;
|
||||
coll_data = osc_count_r + nprocs_world;
|
||||
coll_count = coll_data + nprocs_world;
|
||||
|
||||
size_histogram = coll_count + nprocs_world;
|
||||
}
|
||||
|
||||
/* For all procs in the same MPI_COMM_WORLD we need to add them to the hash table */
|
||||
for( i = 0; i < nprocs; i++ ) {
|
||||
|
||||
/* Extract the peer procname from the procs array */
|
||||
if( ompi_proc_is_sentinel(procs[i]) ) {
|
||||
tmp = ompi_proc_sentinel_to_name((uintptr_t)procs[i]);
|
||||
} else {
|
||||
tmp = procs[i]->super.proc_name;
|
||||
}
|
||||
if( tmp.jobid != ompi_proc_local_proc->super.proc_name.jobid )
|
||||
continue;
|
||||
|
||||
/* each process will only be added once, so there is no way it already exists in the hash */
|
||||
for( peer_rank = 0; peer_rank < nprocs_world; peer_rank++ ) {
|
||||
wp_name = ompi_group_get_proc_name(((ompi_communicator_t*)&ompi_mpi_comm_world)->c_remote_group, peer_rank);
|
||||
if( 0 != opal_compare_proc( tmp, wp_name ) )
|
||||
continue;
|
||||
|
||||
key = *((uint64_t*)&tmp);
|
||||
/* save the rank of the process in MPI_COMM_WORLD in the hash using the proc_name as the key */
|
||||
if( OPAL_SUCCESS != opal_hash_table_set_value_uint64(common_monitoring_translation_ht,
|
||||
key, (void*)(uintptr_t)peer_rank) ) {
|
||||
return OMPI_ERR_OUT_OF_RESOURCE; /* failed to allocate memory or growing the hash table */
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static void mca_common_monitoring_reset( void )
|
||||
{
|
||||
int array_size = (10 + max_size_histogram) * nprocs_world;
|
||||
memset(pml_data, 0, array_size * sizeof(size_t));
|
||||
mca_common_monitoring_coll_reset();
|
||||
}
|
||||
|
||||
void mca_common_monitoring_record_pml(int world_rank, size_t data_size, int tag)
|
||||
{
|
||||
if( 0 == mca_common_monitoring_current_state ) return; /* right now the monitoring is not started */
|
||||
|
||||
/* Keep tracks of the data_size distribution */
|
||||
if( 0 == data_size ) {
|
||||
opal_atomic_add_size_t(&size_histogram[world_rank * max_size_histogram], 1);
|
||||
} else {
|
||||
int log2_size = log10(data_size)/log10_2;
|
||||
if(log2_size > max_size_histogram - 2) /* Avoid out-of-bound write */
|
||||
log2_size = max_size_histogram - 2;
|
||||
opal_atomic_add_size_t(&size_histogram[world_rank * max_size_histogram + log2_size + 1], 1);
|
||||
}
|
||||
|
||||
/* distinguishses positive and negative tags if requested */
|
||||
if( (tag < 0) && (mca_common_monitoring_filter()) ) {
|
||||
opal_atomic_add_size_t(&filtered_pml_data[world_rank], data_size);
|
||||
opal_atomic_add_size_t(&filtered_pml_count[world_rank], 1);
|
||||
} else { /* if filtered monitoring is not activated data is aggregated indifferently */
|
||||
opal_atomic_add_size_t(&pml_data[world_rank], data_size);
|
||||
opal_atomic_add_size_t(&pml_count[world_rank], 1);
|
||||
}
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_pml_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int i, comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_count)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = pml_count[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_pml_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
int i;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_data)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = pml_data[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
void mca_common_monitoring_record_osc(int world_rank, size_t data_size,
|
||||
enum mca_monitoring_osc_direction dir)
|
||||
{
|
||||
if( 0 == mca_common_monitoring_current_state ) return; /* right now the monitoring is not started */
|
||||
|
||||
if( SEND == dir ) {
|
||||
opal_atomic_add_size_t(&osc_data_s[world_rank], data_size);
|
||||
opal_atomic_add_size_t(&osc_count_s[world_rank], 1);
|
||||
} else {
|
||||
opal_atomic_add_size_t(&osc_data_r[world_rank], data_size);
|
||||
opal_atomic_add_size_t(&osc_count_r[world_rank], 1);
|
||||
}
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_osc_sent_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int i, comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_count)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = osc_count_s[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_osc_sent_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
int i;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_data)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = osc_data_s[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_osc_recv_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int i, comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_count)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = osc_count_r[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_osc_recv_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
int i;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_data)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = osc_data_r[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
void mca_common_monitoring_record_coll(int world_rank, size_t data_size)
|
||||
{
|
||||
if( 0 == mca_common_monitoring_current_state ) return; /* right now the monitoring is not started */
|
||||
|
||||
opal_atomic_add_size_t(&coll_data[world_rank], data_size);
|
||||
opal_atomic_add_size_t(&coll_count[world_rank], 1);
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_coll_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int i, comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_count)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = coll_count[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_common_monitoring_get_coll_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int comm_size = ompi_comm_size (comm);
|
||||
size_t *values = (size_t*) value;
|
||||
int i;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == pml_data)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = coll_data[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static void mca_common_monitoring_output( FILE *pf, int my_rank, int nbprocs )
|
||||
{
|
||||
/* Dump outgoing messages */
|
||||
fprintf(pf, "# POINT TO POINT\n");
|
||||
for (int i = 0 ; i < nbprocs ; i++) {
|
||||
if(pml_count[i] > 0) {
|
||||
fprintf(pf, "E\t%" PRId32 "\t%" PRId32 "\t%zu bytes\t%zu msgs sent\t",
|
||||
my_rank, i, pml_data[i], pml_count[i]);
|
||||
for(int j = 0 ; j < max_size_histogram ; ++j)
|
||||
fprintf(pf, "%zu%s", size_histogram[i * max_size_histogram + j],
|
||||
j < max_size_histogram - 1 ? "," : "\n");
|
||||
}
|
||||
}
|
||||
|
||||
/* Dump outgoing synchronization/collective messages */
|
||||
if( mca_common_monitoring_filter() ) {
|
||||
for (int i = 0 ; i < nbprocs ; i++) {
|
||||
if(filtered_pml_count[i] > 0) {
|
||||
fprintf(pf, "I\t%" PRId32 "\t%" PRId32 "\t%zu bytes\t%zu msgs sent%s",
|
||||
my_rank, i, filtered_pml_data[i], filtered_pml_count[i],
|
||||
0 == pml_count[i] ? "\t" : "\n");
|
||||
/*
|
||||
* In the case there was no external messages
|
||||
* exchanged between the two processes, the histogram
|
||||
* has not yet been dumpped. Then we need to add it at
|
||||
* the end of the internal category.
|
||||
*/
|
||||
if(0 == pml_count[i]) {
|
||||
for(int j = 0 ; j < max_size_histogram ; ++j)
|
||||
fprintf(pf, "%zu%s", size_histogram[i * max_size_histogram + j],
|
||||
j < max_size_histogram - 1 ? "," : "\n");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Dump incoming messages */
|
||||
fprintf(pf, "# OSC\n");
|
||||
for (int i = 0 ; i < nbprocs ; i++) {
|
||||
if(osc_count_s[i] > 0) {
|
||||
fprintf(pf, "S\t%" PRId32 "\t%" PRId32 "\t%zu bytes\t%zu msgs sent\n",
|
||||
my_rank, i, osc_data_s[i], osc_count_s[i]);
|
||||
}
|
||||
if(osc_count_r[i] > 0) {
|
||||
fprintf(pf, "R\t%" PRId32 "\t%" PRId32 "\t%zu bytes\t%zu msgs sent\n",
|
||||
my_rank, i, osc_data_r[i], osc_count_r[i]);
|
||||
}
|
||||
}
|
||||
|
||||
/* Dump collectives */
|
||||
fprintf(pf, "# COLLECTIVES\n");
|
||||
for (int i = 0 ; i < nbprocs ; i++) {
|
||||
if(coll_count[i] > 0) {
|
||||
fprintf(pf, "C\t%" PRId32 "\t%" PRId32 "\t%zu bytes\t%zu msgs sent\n",
|
||||
my_rank, i, coll_data[i], coll_count[i]);
|
||||
}
|
||||
}
|
||||
mca_common_monitoring_coll_flush_all(pf);
|
||||
}
|
||||
|
||||
/*
|
||||
* Flushes the monitoring into filename
|
||||
* Useful for phases (see example in test/monitoring)
|
||||
*/
|
||||
static int mca_common_monitoring_flush(int fd, char* filename)
|
||||
{
|
||||
/* If we are not drived by MPIT then dump the monitoring information */
|
||||
if( 0 == mca_common_monitoring_current_state || 0 == fd ) /* if disabled do nothing */
|
||||
return OMPI_SUCCESS;
|
||||
|
||||
if( 1 == fd ) {
|
||||
OPAL_MONITORING_PRINT_INFO("Proc %" PRId32 " flushing monitoring to stdout", rank_world);
|
||||
mca_common_monitoring_output( stdout, rank_world, nprocs_world );
|
||||
} else if( 2 == fd ) {
|
||||
OPAL_MONITORING_PRINT_INFO("Proc %" PRId32 " flushing monitoring to stderr", rank_world);
|
||||
mca_common_monitoring_output( stderr, rank_world, nprocs_world );
|
||||
} else {
|
||||
FILE *pf = NULL;
|
||||
char* tmpfn = NULL;
|
||||
|
||||
if( NULL == filename ) { /* No filename */
|
||||
OPAL_MONITORING_PRINT_ERR("Error while flushing: no filename provided");
|
||||
return OMPI_ERROR;
|
||||
} else {
|
||||
asprintf(&tmpfn, "%s.%" PRId32 ".prof", filename, rank_world);
|
||||
pf = fopen(tmpfn, "w");
|
||||
free(tmpfn);
|
||||
}
|
||||
|
||||
if(NULL == pf) { /* Error during open */
|
||||
OPAL_MONITORING_PRINT_ERR("Error while flushing to: %s.%" PRId32 ".prof",
|
||||
filename, rank_world);
|
||||
return OMPI_ERROR;
|
||||
}
|
||||
|
||||
OPAL_MONITORING_PRINT_INFO("Proc %d flushing monitoring to: %s.%" PRId32 ".prof",
|
||||
rank_world, filename, rank_world);
|
||||
|
||||
mca_common_monitoring_output( pf, rank_world, nprocs_world );
|
||||
|
||||
fclose(pf);
|
||||
}
|
||||
/* Reset to 0 all monitored data */
|
||||
mca_common_monitoring_reset();
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
120
ompi/mca/common/monitoring/common_monitoring.h
Обычный файл
120
ompi/mca/common/monitoring/common_monitoring.h
Обычный файл
@ -0,0 +1,120 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_COMMON_MONITORING_H
|
||||
#define MCA_COMMON_MONITORING_H
|
||||
|
||||
BEGIN_C_DECLS
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/proc/proc.h>
|
||||
#include <ompi/group/group.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <opal/class/opal_hash_table.h>
|
||||
#include <opal/mca/base/mca_base_pvar.h>
|
||||
|
||||
#define MCA_MONITORING_MAKE_VERSION \
|
||||
MCA_BASE_MAKE_VERSION(component, OMPI_MAJOR_VERSION, OMPI_MINOR_VERSION, OMPI_RELEASE_VERSION)
|
||||
|
||||
#define OPAL_MONITORING_VERBOSE(x, ...) \
|
||||
OPAL_OUTPUT_VERBOSE((x, mca_common_monitoring_output_stream_id, __VA_ARGS__))
|
||||
|
||||
/* When built in debug mode, always display error messages */
|
||||
#if OPAL_ENABLE_DEBUG
|
||||
#define OPAL_MONITORING_PRINT_ERR(...) \
|
||||
OPAL_MONITORING_VERBOSE(0, __VA_ARGS__)
|
||||
#else /* if( ! OPAL_ENABLE_DEBUG ) */
|
||||
#define OPAL_MONITORING_PRINT_ERR(...) \
|
||||
OPAL_MONITORING_VERBOSE(1, __VA_ARGS__)
|
||||
#endif /* OPAL_ENABLE_DEBUG */
|
||||
|
||||
#define OPAL_MONITORING_PRINT_WARN(...) \
|
||||
OPAL_MONITORING_VERBOSE(5, __VA_ARGS__)
|
||||
|
||||
#define OPAL_MONITORING_PRINT_INFO(...) \
|
||||
OPAL_MONITORING_VERBOSE(10, __VA_ARGS__)
|
||||
|
||||
extern int mca_common_monitoring_output_stream_id;
|
||||
extern int mca_common_monitoring_enabled;
|
||||
extern int mca_common_monitoring_current_state;
|
||||
extern opal_hash_table_t *common_monitoring_translation_ht;
|
||||
|
||||
OMPI_DECLSPEC void mca_common_monitoring_register(void*pml_monitoring_component);
|
||||
OMPI_DECLSPEC void mca_common_monitoring_init( void );
|
||||
OMPI_DECLSPEC void mca_common_monitoring_finalize( void );
|
||||
OMPI_DECLSPEC int mca_common_monitoring_add_procs(struct ompi_proc_t **procs, size_t nprocs);
|
||||
|
||||
/* Records PML communication */
|
||||
OMPI_DECLSPEC void mca_common_monitoring_record_pml(int world_rank, size_t data_size, int tag);
|
||||
|
||||
/* SEND corresponds to data emitted from the current proc to the given
|
||||
* one. RECV represents data emitted from the given proc to the
|
||||
* current one.
|
||||
*/
|
||||
enum mca_monitoring_osc_direction { SEND, RECV };
|
||||
|
||||
/* Records OSC communications. */
|
||||
OMPI_DECLSPEC void mca_common_monitoring_record_osc(int world_rank, size_t data_size,
|
||||
enum mca_monitoring_osc_direction dir);
|
||||
|
||||
/* Records COLL communications. */
|
||||
OMPI_DECLSPEC void mca_common_monitoring_record_coll(int world_rank, size_t data_size);
|
||||
|
||||
/* Translate the rank from the given communicator of a process to its rank in MPI_COMM_RANK. */
|
||||
static inline int mca_common_monitoring_get_world_rank(int dst, struct ompi_communicator_t*comm,
|
||||
int*world_rank)
|
||||
{
|
||||
opal_process_name_t tmp;
|
||||
|
||||
/* find the processor of the destination */
|
||||
ompi_proc_t *proc = ompi_group_get_proc_ptr(comm->c_remote_group, dst, true);
|
||||
if( ompi_proc_is_sentinel(proc) ) {
|
||||
tmp = ompi_proc_sentinel_to_name((uintptr_t)proc);
|
||||
} else {
|
||||
tmp = proc->super.proc_name;
|
||||
}
|
||||
|
||||
/* find its name*/
|
||||
uint64_t rank, key = *((uint64_t*)&tmp);
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
int ret = opal_hash_table_get_value_uint64(common_monitoring_translation_ht,
|
||||
key, (void *)&rank);
|
||||
|
||||
/* Use intermediate variable to avoid overwriting while looking up in the hashtbale. */
|
||||
if( ret == OPAL_SUCCESS ) *world_rank = (int)rank;
|
||||
return ret;
|
||||
}
|
||||
|
||||
/* Return the current status of the monitoring system 0 if off or the
|
||||
* seperation between internal tags and external tags is disabled. Any
|
||||
* other positive value if the segregation between point-to-point and
|
||||
* collective is enabled.
|
||||
*/
|
||||
static inline int mca_common_monitoring_filter( void )
|
||||
{
|
||||
return 1 < mca_common_monitoring_current_state;
|
||||
}
|
||||
|
||||
/* Collective operation monitoring */
|
||||
struct mca_monitoring_coll_data_t;
|
||||
typedef struct mca_monitoring_coll_data_t mca_monitoring_coll_data_t;
|
||||
OMPI_DECLSPEC OBJ_CLASS_DECLARATION(mca_monitoring_coll_data_t);
|
||||
|
||||
OMPI_DECLSPEC mca_monitoring_coll_data_t*mca_common_monitoring_coll_new(ompi_communicator_t*comm);
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_release(mca_monitoring_coll_data_t*data);
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_o2a(size_t size, mca_monitoring_coll_data_t*data);
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_a2o(size_t size, mca_monitoring_coll_data_t*data);
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_a2a(size_t size, mca_monitoring_coll_data_t*data);
|
||||
|
||||
END_C_DECLS
|
||||
|
||||
#endif /* MCA_COMMON_MONITORING_H */
|
||||
380
ompi/mca/common/monitoring/common_monitoring_coll.c
Обычный файл
380
ompi/mca/common/monitoring/common_monitoring_coll.c
Обычный файл
@ -0,0 +1,380 @@
|
||||
/*
|
||||
* Copyright (c) 2013-2016 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2015 Bull SAS. All rights reserved.
|
||||
* Copyright (c) 2016-2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include "common_monitoring.h"
|
||||
#include "common_monitoring_coll.h"
|
||||
#include <ompi/constants.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <opal/mca/base/mca_base_component_repository.h>
|
||||
#include <opal/class/opal_hash_table.h>
|
||||
#include <assert.h>
|
||||
|
||||
/*** Monitoring specific variables ***/
|
||||
struct mca_monitoring_coll_data_t {
|
||||
opal_object_t super;
|
||||
char*procs;
|
||||
char*comm_name;
|
||||
int world_rank;
|
||||
int is_released;
|
||||
ompi_communicator_t*p_comm;
|
||||
size_t o2a_count;
|
||||
size_t o2a_size;
|
||||
size_t a2o_count;
|
||||
size_t a2o_size;
|
||||
size_t a2a_count;
|
||||
size_t a2a_size;
|
||||
};
|
||||
|
||||
/* Collectives operation monitoring */
|
||||
static opal_hash_table_t *comm_data = NULL;
|
||||
|
||||
/* Check whether the communicator's name have been changed. Update the
|
||||
* data->comm_name field if so.
|
||||
*/
|
||||
static inline void mca_common_monitoring_coll_check_name(mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
if( data->comm_name && data->p_comm && (data->p_comm->c_flags & OMPI_COMM_NAMEISSET)
|
||||
&& data->p_comm->c_name && 0 < strlen(data->p_comm->c_name)
|
||||
&& 0 != strncmp(data->p_comm->c_name, data->comm_name, OPAL_MAX_OBJECT_NAME - 1) )
|
||||
{
|
||||
free(data->comm_name);
|
||||
data->comm_name = strdup(data->p_comm->c_name);
|
||||
}
|
||||
}
|
||||
|
||||
static inline void mca_common_monitoring_coll_cache(mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
int world_rank;
|
||||
if( NULL == data->comm_name && 0 < strlen(data->p_comm->c_name) ) {
|
||||
data->comm_name = strdup(data->p_comm->c_name);
|
||||
} else {
|
||||
mca_common_monitoring_coll_check_name(data);
|
||||
}
|
||||
if( -1 == data->world_rank ) {
|
||||
/* Get current process world_rank */
|
||||
mca_common_monitoring_get_world_rank(ompi_comm_rank(data->p_comm), data->p_comm,
|
||||
&data->world_rank);
|
||||
}
|
||||
/* Only list procs if the hashtable is already initialized, ie if the previous call worked */
|
||||
if( (-1 != data->world_rank) && (NULL == data->procs || 0 == strlen(data->procs)) ) {
|
||||
int i, pos = 0, size, world_size = -1, max_length;
|
||||
char*tmp_procs;
|
||||
size = ompi_comm_size(data->p_comm);
|
||||
world_size = ompi_comm_size((ompi_communicator_t*)&ompi_mpi_comm_world) - 1;
|
||||
assert( 0 < size );
|
||||
/* Allocate enough space for list (add 1 to keep the final '\0' if already exact size) */
|
||||
max_length = snprintf(NULL, 0, "%d,", world_size - 1) + 1;
|
||||
tmp_procs = malloc((1 + max_length * size) * sizeof(char));
|
||||
if( NULL == tmp_procs ) {
|
||||
OPAL_MONITORING_PRINT_ERR("Cannot allocate memory for caching proc list.");
|
||||
} else {
|
||||
tmp_procs[0] = '\0';
|
||||
/* Build procs list */
|
||||
for(i = 0; i < size; ++i) {
|
||||
mca_common_monitoring_get_world_rank(i, data->p_comm, &world_rank);
|
||||
pos += sprintf(&tmp_procs[pos], "%d,", world_rank);
|
||||
}
|
||||
tmp_procs[pos - 1] = '\0'; /* Remove final coma */
|
||||
data->procs = realloc(tmp_procs, pos * sizeof(char)); /* Adjust to size required */
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mca_monitoring_coll_data_t*mca_common_monitoring_coll_new( ompi_communicator_t*comm )
|
||||
{
|
||||
mca_monitoring_coll_data_t*data = OBJ_NEW(mca_monitoring_coll_data_t);
|
||||
if( NULL == data ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: new: data structure cannot be allocated");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
data->p_comm = comm;
|
||||
|
||||
/* Allocate hashtable */
|
||||
if( NULL == comm_data ) {
|
||||
comm_data = OBJ_NEW(opal_hash_table_t);
|
||||
if( NULL == comm_data ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: new: failed to allocate hashtable");
|
||||
return data;
|
||||
}
|
||||
opal_hash_table_init(comm_data, 2048);
|
||||
}
|
||||
|
||||
/* Insert in hashtable */
|
||||
uint64_t key = *((uint64_t*)&comm);
|
||||
if( OPAL_SUCCESS != opal_hash_table_set_value_uint64(comm_data, key, (void*)data) ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: new: failed to allocate memory or "
|
||||
"growing the hash table");
|
||||
}
|
||||
|
||||
/* Cache data so the procs can be released without affecting the output */
|
||||
mca_common_monitoring_coll_cache(data);
|
||||
|
||||
return data;
|
||||
}
|
||||
|
||||
void mca_common_monitoring_coll_release(mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
#if OPAL_ENABLE_DEBUG
|
||||
if( NULL == data ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: release: data structure empty or already desallocated");
|
||||
return;
|
||||
}
|
||||
#endif /* OPAL_ENABLE_DEBUG */
|
||||
|
||||
/* not flushed yet */
|
||||
mca_common_monitoring_coll_cache(data);
|
||||
data->is_released = 1;
|
||||
}
|
||||
|
||||
static void mca_common_monitoring_coll_cond_release(mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
#if OPAL_ENABLE_DEBUG
|
||||
if( NULL == data ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: release: data structure empty or already desallocated");
|
||||
return;
|
||||
}
|
||||
#endif /* OPAL_ENABLE_DEBUG */
|
||||
|
||||
if( data->is_released ) { /* if the communicator is already released */
|
||||
opal_hash_table_remove_value_uint64(comm_data, *((uint64_t*)&data->p_comm));
|
||||
data->p_comm = NULL;
|
||||
free(data->comm_name);
|
||||
free(data->procs);
|
||||
OBJ_RELEASE(data);
|
||||
}
|
||||
}
|
||||
|
||||
void mca_common_monitoring_coll_finalize( void )
|
||||
{
|
||||
if( NULL != comm_data ) {
|
||||
opal_hash_table_remove_all( comm_data );
|
||||
OBJ_RELEASE(comm_data);
|
||||
}
|
||||
}
|
||||
|
||||
void mca_common_monitoring_coll_flush(FILE *pf, mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
/* Check for any change in the communicator's name */
|
||||
mca_common_monitoring_coll_check_name(data);
|
||||
|
||||
/* Flush data */
|
||||
fprintf(pf,
|
||||
"D\t%s\tprocs: %s\n"
|
||||
"O2A\t%" PRId32 "\t%zu bytes\t%zu msgs sent\n"
|
||||
"A2O\t%" PRId32 "\t%zu bytes\t%zu msgs sent\n"
|
||||
"A2A\t%" PRId32 "\t%zu bytes\t%zu msgs sent\n",
|
||||
data->comm_name ? data->comm_name : "(no-name)", data->procs,
|
||||
data->world_rank, data->o2a_size, data->o2a_count,
|
||||
data->world_rank, data->a2o_size, data->a2o_count,
|
||||
data->world_rank, data->a2a_size, data->a2a_count);
|
||||
}
|
||||
|
||||
void mca_common_monitoring_coll_flush_all(FILE *pf)
|
||||
{
|
||||
if( NULL == comm_data ) return; /* No hashtable */
|
||||
|
||||
uint64_t key;
|
||||
mca_monitoring_coll_data_t*previous = NULL, *data;
|
||||
|
||||
OPAL_HASH_TABLE_FOREACH(key, uint64, data, comm_data) {
|
||||
if( NULL != previous && NULL == previous->p_comm ) {
|
||||
/* Phase flushed -> free already released once coll_data_t */
|
||||
mca_common_monitoring_coll_cond_release(previous);
|
||||
}
|
||||
mca_common_monitoring_coll_flush(pf, data);
|
||||
previous = data;
|
||||
}
|
||||
mca_common_monitoring_coll_cond_release(previous);
|
||||
}
|
||||
|
||||
|
||||
void mca_common_monitoring_coll_reset(void)
|
||||
{
|
||||
if( NULL == comm_data ) return; /* No hashtable */
|
||||
|
||||
uint64_t key;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
|
||||
OPAL_HASH_TABLE_FOREACH(key, uint64, data, comm_data) {
|
||||
data->o2a_count = 0; data->o2a_size = 0;
|
||||
data->a2o_count = 0; data->a2o_size = 0;
|
||||
data->a2a_count = 0; data->a2a_size = 0;
|
||||
}
|
||||
}
|
||||
|
||||
int mca_common_monitoring_coll_messages_notify(mca_base_pvar_t *pvar,
|
||||
mca_base_pvar_event_t event,
|
||||
void *obj_handle,
|
||||
int *count)
|
||||
{
|
||||
switch (event) {
|
||||
case MCA_BASE_PVAR_HANDLE_BIND:
|
||||
*count = 1;
|
||||
case MCA_BASE_PVAR_HANDLE_UNBIND:
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_START:
|
||||
mca_common_monitoring_current_state = mca_common_monitoring_enabled;
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_STOP:
|
||||
mca_common_monitoring_current_state = 0;
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
return OMPI_ERROR;
|
||||
}
|
||||
|
||||
void mca_common_monitoring_coll_o2a(size_t size, mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
if( 0 == mca_common_monitoring_current_state ) return; /* right now the monitoring is not started */
|
||||
#if OPAL_ENABLE_DEBUG
|
||||
if( NULL == data ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: o2a: data structure empty");
|
||||
return;
|
||||
}
|
||||
#endif /* OPAL_ENABLE_DEBUG */
|
||||
opal_atomic_add_size_t(&data->o2a_size, size);
|
||||
opal_atomic_add_size_t(&data->o2a_count, 1);
|
||||
}
|
||||
|
||||
int mca_common_monitoring_coll_get_o2a_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
size_t *value_size = (size_t*) value;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
int ret = opal_hash_table_get_value_uint64(comm_data, *((uint64_t*)&comm), (void*)&data);
|
||||
if( OPAL_SUCCESS == ret ) {
|
||||
*value_size = data->o2a_count;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
int mca_common_monitoring_coll_get_o2a_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
size_t *value_size = (size_t*) value;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
int ret = opal_hash_table_get_value_uint64(comm_data, *((uint64_t*)&comm), (void*)&data);
|
||||
if( OPAL_SUCCESS == ret ) {
|
||||
*value_size = data->o2a_size;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
void mca_common_monitoring_coll_a2o(size_t size, mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
if( 0 == mca_common_monitoring_current_state ) return; /* right now the monitoring is not started */
|
||||
#if OPAL_ENABLE_DEBUG
|
||||
if( NULL == data ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: a2o: data structure empty");
|
||||
return;
|
||||
}
|
||||
#endif /* OPAL_ENABLE_DEBUG */
|
||||
opal_atomic_add_size_t(&data->a2o_size, size);
|
||||
opal_atomic_add_size_t(&data->a2o_count, 1);
|
||||
}
|
||||
|
||||
int mca_common_monitoring_coll_get_a2o_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
size_t *value_size = (size_t*) value;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
int ret = opal_hash_table_get_value_uint64(comm_data, *((uint64_t*)&comm), (void*)&data);
|
||||
if( OPAL_SUCCESS == ret ) {
|
||||
*value_size = data->a2o_count;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
int mca_common_monitoring_coll_get_a2o_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
size_t *value_size = (size_t*) value;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
int ret = opal_hash_table_get_value_uint64(comm_data, *((uint64_t*)&comm), (void*)&data);
|
||||
if( OPAL_SUCCESS == ret ) {
|
||||
*value_size = data->a2o_size;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
void mca_common_monitoring_coll_a2a(size_t size, mca_monitoring_coll_data_t*data)
|
||||
{
|
||||
if( 0 == mca_common_monitoring_current_state ) return; /* right now the monitoring is not started */
|
||||
#if OPAL_ENABLE_DEBUG
|
||||
if( NULL == data ) {
|
||||
OPAL_MONITORING_PRINT_ERR("coll: a2a: data structure empty");
|
||||
return;
|
||||
}
|
||||
#endif /* OPAL_ENABLE_DEBUG */
|
||||
opal_atomic_add_size_t(&data->a2a_size, size);
|
||||
opal_atomic_add_size_t(&data->a2a_count, 1);
|
||||
}
|
||||
|
||||
int mca_common_monitoring_coll_get_a2a_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
size_t *value_size = (size_t*) value;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
int ret = opal_hash_table_get_value_uint64(comm_data, *((uint64_t*)&comm), (void*)&data);
|
||||
if( OPAL_SUCCESS == ret ) {
|
||||
*value_size = data->a2a_count;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
int mca_common_monitoring_coll_get_a2a_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
size_t *value_size = (size_t*) value;
|
||||
mca_monitoring_coll_data_t*data;
|
||||
int ret = opal_hash_table_get_value_uint64(comm_data, *((uint64_t*)&comm), (void*)&data);
|
||||
if( OPAL_SUCCESS == ret ) {
|
||||
*value_size = data->a2a_size;
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
static void mca_monitoring_coll_construct (mca_monitoring_coll_data_t*coll_data)
|
||||
{
|
||||
coll_data->procs = NULL;
|
||||
coll_data->comm_name = NULL;
|
||||
coll_data->world_rank = -1;
|
||||
coll_data->p_comm = NULL;
|
||||
coll_data->is_released = 0;
|
||||
coll_data->o2a_count = 0;
|
||||
coll_data->o2a_size = 0;
|
||||
coll_data->a2o_count = 0;
|
||||
coll_data->a2o_size = 0;
|
||||
coll_data->a2a_count = 0;
|
||||
coll_data->a2a_size = 0;
|
||||
}
|
||||
|
||||
static void mca_monitoring_coll_destruct (mca_monitoring_coll_data_t*coll_data){}
|
||||
|
||||
OBJ_CLASS_INSTANCE(mca_monitoring_coll_data_t, opal_object_t, mca_monitoring_coll_construct, mca_monitoring_coll_destruct);
|
||||
59
ompi/mca/common/monitoring/common_monitoring_coll.h
Обычный файл
59
ompi/mca/common/monitoring/common_monitoring_coll.h
Обычный файл
@ -0,0 +1,59 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2017 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_COMMON_MONITORING_COLL_H
|
||||
#define MCA_COMMON_MONITORING_COLL_H
|
||||
|
||||
BEGIN_C_DECLS
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <opal/mca/base/mca_base_pvar.h>
|
||||
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_flush(FILE *pf, mca_monitoring_coll_data_t*data);
|
||||
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_flush_all(FILE *pf);
|
||||
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_reset( void );
|
||||
|
||||
OMPI_DECLSPEC int mca_common_monitoring_coll_messages_notify(mca_base_pvar_t *pvar,
|
||||
mca_base_pvar_event_t event,
|
||||
void *obj_handle,
|
||||
int *count);
|
||||
|
||||
OMPI_DECLSPEC int mca_common_monitoring_coll_get_o2a_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
OMPI_DECLSPEC int mca_common_monitoring_coll_get_o2a_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
OMPI_DECLSPEC int mca_common_monitoring_coll_get_a2o_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
OMPI_DECLSPEC int mca_common_monitoring_coll_get_a2o_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
OMPI_DECLSPEC int mca_common_monitoring_coll_get_a2a_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
OMPI_DECLSPEC int mca_common_monitoring_coll_get_a2a_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
OMPI_DECLSPEC void mca_common_monitoring_coll_finalize( void );
|
||||
END_C_DECLS
|
||||
|
||||
#endif /* MCA_COMMON_MONITORING_COLL_H */
|
||||
38
ompi/mca/osc/monitoring/Makefile.am
Обычный файл
38
ompi/mca/osc/monitoring/Makefile.am
Обычный файл
@ -0,0 +1,38 @@
|
||||
#
|
||||
# Copyright (c) 2016 Inria. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
#
|
||||
# $HEADER$
|
||||
#
|
||||
|
||||
monitoring_sources = \
|
||||
osc_monitoring.h \
|
||||
osc_monitoring_comm.h \
|
||||
osc_monitoring_component.c \
|
||||
osc_monitoring_accumulate.h \
|
||||
osc_monitoring_passive_target.h \
|
||||
osc_monitoring_active_target.h \
|
||||
osc_monitoring_dynamic.h \
|
||||
osc_monitoring_module.h \
|
||||
osc_monitoring_template.h
|
||||
|
||||
if MCA_BUILD_ompi_osc_monitoring_DSO
|
||||
component_noinst =
|
||||
component_install = mca_osc_monitoring.la
|
||||
else
|
||||
component_noinst = libmca_osc_monitoring.la
|
||||
component_install =
|
||||
endif
|
||||
|
||||
mcacomponentdir = $(ompilibdir)
|
||||
mcacomponent_LTLIBRARIES = $(component_install)
|
||||
mca_osc_monitoring_la_SOURCES = $(monitoring_sources)
|
||||
mca_osc_monitoring_la_LDFLAGS = -module -avoid-version
|
||||
mca_osc_monitoring_la_LIBADD = \
|
||||
$(OMPI_TOP_BUILDDIR)/ompi/mca/common/monitoring/libmca_common_monitoring.la
|
||||
|
||||
noinst_LTLIBRARIES = $(component_noinst)
|
||||
libmca_osc_monitoring_la_SOURCES = $(monitoring_sources)
|
||||
libmca_osc_monitoring_la_LDFLAGS = -module -avoid-version
|
||||
19
ompi/mca/osc/monitoring/configure.m4
Обычный файл
19
ompi/mca/osc/monitoring/configure.m4
Обычный файл
@ -0,0 +1,19 @@
|
||||
# -*- shell-script -*-
|
||||
#
|
||||
# Copyright (c) 2016 Inria. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
#
|
||||
# $HEADER$
|
||||
#
|
||||
|
||||
# MCA_ompi_osc_monitoring_CONFIG()
|
||||
# ------------------------------------------------
|
||||
AC_DEFUN([MCA_ompi_osc_monitoring_CONFIG],[
|
||||
AC_CONFIG_FILES([ompi/mca/osc/monitoring/Makefile])
|
||||
|
||||
OPAL_CHECK_PORTALS4([osc_monitoring],
|
||||
[AC_DEFINE([OMPI_WITH_OSC_PORTALS4], [1], [Whether or not to generate template for osc_portals4])],
|
||||
[])
|
||||
])dnl
|
||||
29
ompi/mca/osc/monitoring/osc_monitoring.h
Обычный файл
29
ompi/mca/osc/monitoring/osc_monitoring.h
Обычный файл
@ -0,0 +1,29 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_H
|
||||
#define MCA_OSC_MONITORING_H
|
||||
|
||||
BEGIN_C_DECLS
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/mca/osc/osc.h>
|
||||
#include <ompi/mca/common/monitoring/common_monitoring.h>
|
||||
|
||||
struct ompi_osc_monitoring_component_t {
|
||||
ompi_osc_base_component_t super;
|
||||
int priority;
|
||||
};
|
||||
typedef struct ompi_osc_monitoring_component_t ompi_osc_monitoring_component_t;
|
||||
|
||||
OMPI_DECLSPEC extern ompi_osc_monitoring_component_t mca_osc_monitoring_component;
|
||||
|
||||
END_C_DECLS
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_H */
|
||||
175
ompi/mca/osc/monitoring/osc_monitoring_accumulate.h
Обычный файл
175
ompi/mca/osc/monitoring/osc_monitoring_accumulate.h
Обычный файл
@ -0,0 +1,175 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_ACCUMULATE_H
|
||||
#define MCA_OSC_MONITORING_ACCUMULATE_H
|
||||
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/op/op.h>
|
||||
#include <ompi/win/win.h>
|
||||
|
||||
#define OSC_MONITORING_GENERATE_TEMPLATE_ACCUMULATE(template) \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_compare_and_swap (const void *origin_addr, \
|
||||
const void *compare_addr, \
|
||||
void *result_addr, \
|
||||
ompi_datatype_t *dt, \
|
||||
int target_rank, \
|
||||
ptrdiff_t target_disp, \
|
||||
ompi_win_t *win) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size; \
|
||||
ompi_datatype_type_size(dt, &type_size); \
|
||||
mca_common_monitoring_record_osc(world_rank, type_size, SEND); \
|
||||
mca_common_monitoring_record_osc(world_rank, type_size, RECV); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Compare_and_swap to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_compare_and_swap(origin_addr, compare_addr, result_addr, dt, target_rank, target_disp, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_get_accumulate (const void *origin_addr, \
|
||||
int origin_count, \
|
||||
ompi_datatype_t*origin_datatype, \
|
||||
void *result_addr, \
|
||||
int result_count, \
|
||||
ompi_datatype_t*result_datatype, \
|
||||
int target_rank, \
|
||||
MPI_Aint target_disp, \
|
||||
int target_count, \
|
||||
ompi_datatype_t*target_datatype, \
|
||||
ompi_op_t *op, ompi_win_t*win) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, SEND); \
|
||||
ompi_datatype_type_size(result_datatype, &type_size); \
|
||||
data_size = result_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, RECV); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Get_accumulate to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_get_accumulate(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, target_rank, target_disp, target_count, target_datatype, op, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_rget_accumulate (const void *origin_addr, \
|
||||
int origin_count, \
|
||||
ompi_datatype_t *origin_datatype, \
|
||||
void *result_addr, \
|
||||
int result_count, \
|
||||
ompi_datatype_t *result_datatype, \
|
||||
int target_rank, \
|
||||
MPI_Aint target_disp, \
|
||||
int target_count, \
|
||||
ompi_datatype_t*target_datatype, \
|
||||
ompi_op_t *op, \
|
||||
ompi_win_t *win, \
|
||||
ompi_request_t **request) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, SEND); \
|
||||
ompi_datatype_type_size(result_datatype, &type_size); \
|
||||
data_size = result_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, RECV); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Rget_accumulate to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_rget_accumulate(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, target_rank, target_disp, target_count, target_datatype, op, win, request); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_raccumulate (const void *origin_addr, \
|
||||
int origin_count, \
|
||||
ompi_datatype_t *origin_datatype, \
|
||||
int target_rank, \
|
||||
ptrdiff_t target_disp, \
|
||||
int target_count, \
|
||||
ompi_datatype_t *target_datatype, \
|
||||
ompi_op_t *op, ompi_win_t *win, \
|
||||
ompi_request_t **request) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, SEND); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Raccumulate to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_raccumulate(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, win, request); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_accumulate (const void *origin_addr, \
|
||||
int origin_count, \
|
||||
ompi_datatype_t *origin_datatype, \
|
||||
int target_rank, \
|
||||
ptrdiff_t target_disp, \
|
||||
int target_count, \
|
||||
ompi_datatype_t *target_datatype, \
|
||||
ompi_op_t *op, ompi_win_t *win) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, SEND); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Accumulate to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_accumulate(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_fetch_and_op (const void *origin_addr, \
|
||||
void *result_addr, \
|
||||
ompi_datatype_t *dt, \
|
||||
int target_rank, \
|
||||
ptrdiff_t target_disp, \
|
||||
ompi_op_t *op, ompi_win_t *win) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size; \
|
||||
ompi_datatype_type_size(dt, &type_size); \
|
||||
mca_common_monitoring_record_osc(world_rank, type_size, SEND); \
|
||||
mca_common_monitoring_record_osc(world_rank, type_size, RECV); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Fetch_and_op to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_fetch_and_op(origin_addr, result_addr, dt, target_rank, target_disp, op, win); \
|
||||
}
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_ACCUMULATE_H */
|
||||
48
ompi/mca/osc/monitoring/osc_monitoring_active_target.h
Обычный файл
48
ompi/mca/osc/monitoring/osc_monitoring_active_target.h
Обычный файл
@ -0,0 +1,48 @@
|
||||
/*
|
||||
* Copyright (c) 2016 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_ACTIVE_TARGET_H
|
||||
#define MCA_OSC_MONITORING_ACTIVE_TARGET_H
|
||||
|
||||
#include <ompi/group/group.h>
|
||||
#include <ompi/win/win.h>
|
||||
|
||||
#define OSC_MONITORING_GENERATE_TEMPLATE_ACTIVE_TARGET(template) \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_post (ompi_group_t *group, int assert, ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_post(group, assert, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_start (ompi_group_t *group, int assert, ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_start(group, assert, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_complete (ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_complete(win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_wait (ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_wait(win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_test (ompi_win_t *win, int *flag) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_test(win, flag); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_fence (int assert, ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_fence(assert, win); \
|
||||
}
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_ACTIVE_TARGET_H */
|
||||
118
ompi/mca/osc/monitoring/osc_monitoring_comm.h
Обычный файл
118
ompi/mca/osc/monitoring/osc_monitoring_comm.h
Обычный файл
@ -0,0 +1,118 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_COMM_H
|
||||
#define MCA_OSC_MONITORING_COMM_H
|
||||
|
||||
#include <ompi/request/request.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/win/win.h>
|
||||
|
||||
#define OSC_MONITORING_GENERATE_TEMPLATE_COMM(template) \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_put (const void *origin_addr, \
|
||||
int origin_count, \
|
||||
ompi_datatype_t *origin_datatype, \
|
||||
int target_rank, \
|
||||
ptrdiff_t target_disp, \
|
||||
int target_count, \
|
||||
ompi_datatype_t *target_datatype, \
|
||||
ompi_win_t *win) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, SEND); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Put to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_put(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_rput (const void *origin_addr, \
|
||||
int origin_count, \
|
||||
ompi_datatype_t *origin_datatype, \
|
||||
int target_rank, \
|
||||
ptrdiff_t target_disp, \
|
||||
int target_count, \
|
||||
ompi_datatype_t *target_datatype, \
|
||||
ompi_win_t *win, \
|
||||
ompi_request_t **request) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(target_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, SEND); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Rput to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_rput(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win, request); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_get (void *origin_addr, int origin_count, \
|
||||
ompi_datatype_t *origin_datatype, \
|
||||
int source_rank, \
|
||||
ptrdiff_t source_disp, \
|
||||
int source_count, \
|
||||
ompi_datatype_t *source_datatype, \
|
||||
ompi_win_t *win) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(source_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, 0, SEND); \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, RECV); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Get to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_get(origin_addr, origin_count, origin_datatype, source_rank, source_disp, source_count, source_datatype, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_rget (void *origin_addr, int origin_count, \
|
||||
ompi_datatype_t *origin_datatype, \
|
||||
int source_rank, \
|
||||
ptrdiff_t source_disp, \
|
||||
int source_count, \
|
||||
ompi_datatype_t *source_datatype, \
|
||||
ompi_win_t *win, \
|
||||
ompi_request_t **request) \
|
||||
{ \
|
||||
int world_rank; \
|
||||
/** \
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD \
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank \
|
||||
*/ \
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(source_rank, ompi_osc_monitoring_## template ##_get_comm(win), &world_rank)) { \
|
||||
size_t type_size, data_size; \
|
||||
ompi_datatype_type_size(origin_datatype, &type_size); \
|
||||
data_size = origin_count*type_size; \
|
||||
mca_common_monitoring_record_osc(world_rank, 0, SEND); \
|
||||
mca_common_monitoring_record_osc(world_rank, data_size, RECV); \
|
||||
OPAL_MONITORING_PRINT_INFO("MPI_Rget to %d intercepted", world_rank); \
|
||||
} \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_rget(origin_addr, origin_count, origin_datatype, source_rank, source_disp, source_count, source_datatype, win, request); \
|
||||
}
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_COMM_H */
|
||||
|
||||
154
ompi/mca/osc/monitoring/osc_monitoring_component.c
Обычный файл
154
ompi/mca/osc/monitoring/osc_monitoring_component.c
Обычный файл
@ -0,0 +1,154 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include "osc_monitoring.h"
|
||||
#include <ompi/constants.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/info/info.h>
|
||||
#include <ompi/win/win.h>
|
||||
#include <ompi/info/info.h>
|
||||
#include <ompi/mca/osc/osc.h>
|
||||
#include <ompi/mca/osc/base/base.h>
|
||||
#include <opal/mca/base/mca_base_component_repository.h>
|
||||
|
||||
/***************************************/
|
||||
/* Include template generating macros */
|
||||
#include "osc_monitoring_template.h"
|
||||
|
||||
#include <ompi/mca/osc/rdma/osc_rdma.h>
|
||||
OSC_MONITORING_MODULE_TEMPLATE_GENERATE(rdma, ompi_osc_rdma_module_t, comm)
|
||||
#undef GET_MODULE
|
||||
|
||||
#include <ompi/mca/osc/sm/osc_sm.h>
|
||||
OSC_MONITORING_MODULE_TEMPLATE_GENERATE(sm, ompi_osc_sm_module_t, comm)
|
||||
#undef GET_MODULE
|
||||
|
||||
#include <ompi/mca/osc/pt2pt/osc_pt2pt.h>
|
||||
OSC_MONITORING_MODULE_TEMPLATE_GENERATE(pt2pt, ompi_osc_pt2pt_module_t, comm)
|
||||
#undef GET_MODULE
|
||||
|
||||
#ifdef OMPI_WITH_OSC_PORTALS4
|
||||
#include <ompi/mca/osc/portals4/osc_portals4.h>
|
||||
OSC_MONITORING_MODULE_TEMPLATE_GENERATE(portals4, ompi_osc_portals4_module_t, comm)
|
||||
#undef GET_MODULE
|
||||
#endif /* OMPI_WITH_OSC_PORTALS4 */
|
||||
|
||||
/***************************************/
|
||||
|
||||
static int mca_osc_monitoring_component_init(bool enable_progress_threads,
|
||||
bool enable_mpi_threads)
|
||||
{
|
||||
OPAL_MONITORING_PRINT_INFO("osc_component_init");
|
||||
mca_common_monitoring_init();
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_osc_monitoring_component_finish(void)
|
||||
{
|
||||
OPAL_MONITORING_PRINT_INFO("osc_component_finish");
|
||||
mca_common_monitoring_finalize();
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_osc_monitoring_component_register(void)
|
||||
{
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_osc_monitoring_component_query(struct ompi_win_t *win, void **base, size_t size, int disp_unit,
|
||||
struct ompi_communicator_t *comm, struct opal_info_t *info,
|
||||
int flavor)
|
||||
{
|
||||
OPAL_MONITORING_PRINT_INFO("osc_component_query");
|
||||
return mca_osc_monitoring_component.priority;
|
||||
}
|
||||
|
||||
static int mca_osc_monitoring_component_select(struct ompi_win_t *win, void **base, size_t size, int disp_unit,
|
||||
struct ompi_communicator_t *comm, struct opal_info_t *info,
|
||||
int flavor, int *model)
|
||||
{
|
||||
OPAL_MONITORING_PRINT_INFO("osc_component_select");
|
||||
opal_list_item_t *item;
|
||||
ompi_osc_base_component_t *best_component = NULL;
|
||||
int best_priority = -1, priority, ret = OMPI_SUCCESS;
|
||||
|
||||
/* Redo the select loop to add our layer in the middle */
|
||||
for (item = opal_list_get_first(&ompi_osc_base_framework.framework_components) ;
|
||||
item != opal_list_get_end(&ompi_osc_base_framework.framework_components) ;
|
||||
item = opal_list_get_next(item)) {
|
||||
ompi_osc_base_component_t *component = (ompi_osc_base_component_t*)
|
||||
((mca_base_component_list_item_t*) item)->cli_component;
|
||||
|
||||
if( component == (ompi_osc_base_component_t*)(&mca_osc_monitoring_component) )
|
||||
continue; /* skip self */
|
||||
|
||||
priority = component->osc_query(win, base, size, disp_unit, comm, info, flavor);
|
||||
if (priority < 0) {
|
||||
if (MPI_WIN_FLAVOR_SHARED == flavor && OMPI_ERR_RMA_SHARED == priority) {
|
||||
/* NTH: quick fix to return OMPI_ERR_RMA_SHARED */
|
||||
return OMPI_ERR_RMA_SHARED;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (priority > best_priority) {
|
||||
best_component = component;
|
||||
best_priority = priority;
|
||||
}
|
||||
}
|
||||
|
||||
if (NULL == best_component) return OMPI_ERR_NOT_SUPPORTED;
|
||||
OPAL_MONITORING_PRINT_INFO("osc: chosen one: %s", best_component->osc_version.mca_component_name);
|
||||
ret = best_component->osc_select(win, base, size, disp_unit, comm, info, flavor, model);
|
||||
if( OMPI_SUCCESS == ret ) {
|
||||
/* Intercept module functions with ours, based on selected component */
|
||||
if( 0 == strcmp("rdma", best_component->osc_version.mca_component_name) ) {
|
||||
OSC_MONITORING_SET_TEMPLATE(rdma, win->w_osc_module);
|
||||
} else if( 0 == strcmp("sm", best_component->osc_version.mca_component_name) ) {
|
||||
OSC_MONITORING_SET_TEMPLATE(sm, win->w_osc_module);
|
||||
} else if( 0 == strcmp("pt2pt", best_component->osc_version.mca_component_name) ) {
|
||||
OSC_MONITORING_SET_TEMPLATE(pt2pt, win->w_osc_module);
|
||||
#ifdef OMPI_WITH_OSC_PORTALS4
|
||||
} else if( 0 == strcmp("portals4", best_component->osc_version.mca_component_name) ) {
|
||||
OSC_MONITORING_SET_TEMPLATE(portals4, win->w_osc_module);
|
||||
#endif /* OMPI_WITH_OSC_PORTALS4 */
|
||||
} else {
|
||||
OPAL_MONITORING_PRINT_WARN("osc: monitoring disabled: no module for this component "
|
||||
"(%s)", best_component->osc_version.mca_component_name);
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
ompi_osc_monitoring_component_t mca_osc_monitoring_component = {
|
||||
.super = {
|
||||
/* First, the mca_base_component_t struct containing meta
|
||||
information about the component itself */
|
||||
.osc_version = {
|
||||
OMPI_OSC_BASE_VERSION_3_0_0,
|
||||
|
||||
.mca_component_name = "monitoring", /* MCA component name */
|
||||
MCA_MONITORING_MAKE_VERSION,
|
||||
.mca_register_component_params = mca_osc_monitoring_component_register
|
||||
},
|
||||
.osc_data = {
|
||||
/* The component is checkpoint ready */
|
||||
MCA_BASE_METADATA_PARAM_CHECKPOINT
|
||||
},
|
||||
|
||||
.osc_init = mca_osc_monitoring_component_init, /* component init */
|
||||
.osc_finalize = mca_osc_monitoring_component_finish, /* component finalize */
|
||||
.osc_query = mca_osc_monitoring_component_query,
|
||||
.osc_select = mca_osc_monitoring_component_select
|
||||
},
|
||||
.priority = INT_MAX
|
||||
};
|
||||
|
||||
27
ompi/mca/osc/monitoring/osc_monitoring_dynamic.h
Обычный файл
27
ompi/mca/osc/monitoring/osc_monitoring_dynamic.h
Обычный файл
@ -0,0 +1,27 @@
|
||||
/*
|
||||
* Copyright (c) 2016 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_DYNAMIC_H
|
||||
#define MCA_OSC_MONITORING_DYNAMIC_H
|
||||
|
||||
#include <ompi/win/win.h>
|
||||
|
||||
#define OSC_MONITORING_GENERATE_TEMPLATE_DYNAMIC(template) \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_attach (struct ompi_win_t *win, void *base, size_t len) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_win_attach(win, base, len); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_detach (struct ompi_win_t *win, const void *base) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_win_detach(win, base); \
|
||||
}
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_DYNAMIC_H */
|
||||
89
ompi/mca/osc/monitoring/osc_monitoring_module.h
Обычный файл
89
ompi/mca/osc/monitoring/osc_monitoring_module.h
Обычный файл
@ -0,0 +1,89 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_MODULE_H
|
||||
#define MCA_OSC_MONITORING_MODULE_H
|
||||
|
||||
#include <ompi/info/info.h>
|
||||
#include <ompi/win/win.h>
|
||||
#include <ompi/mca/osc/osc.h>
|
||||
|
||||
/* Define once and for all the module_template variable name */
|
||||
#define OMPI_OSC_MONITORING_MODULE_VARIABLE(template) \
|
||||
ompi_osc_monitoring_module_## template ##_template
|
||||
|
||||
/* Define once and for all the module_template variable name */
|
||||
#define OMPI_OSC_MONITORING_MODULE_INIT(template) \
|
||||
ompi_osc_monitoring_module_## template ##_init_done
|
||||
|
||||
/* Define once and for all the template variable name */
|
||||
#define OMPI_OSC_MONITORING_TEMPLATE_VARIABLE(template) \
|
||||
ompi_osc_monitoring_## template ##_template
|
||||
|
||||
/* Define the ompi_osc_monitoring_module_## template ##_template variable */
|
||||
#define OMPI_OSC_MONITORING_MODULE_GENERATE(template) \
|
||||
static ompi_osc_base_module_t OMPI_OSC_MONITORING_MODULE_VARIABLE(template)
|
||||
|
||||
/* Define the ompi_osc_monitoring_module_## template ##_init_done variable */
|
||||
#define OMPI_OSC_MONITORING_MODULE_INIT_GENERATE(template) \
|
||||
static int64_t OMPI_OSC_MONITORING_MODULE_INIT(template)
|
||||
|
||||
/* Define and set the ompi_osc_monitoring_## template ##_template
|
||||
* variable. The functions recorded here are linked to the original
|
||||
* functions of the original {template} module that were replaced.
|
||||
*/
|
||||
#define MCA_OSC_MONITORING_MODULE_TEMPLATE_GENERATE(template) \
|
||||
static ompi_osc_base_module_t OMPI_OSC_MONITORING_TEMPLATE_VARIABLE(template) = { \
|
||||
.osc_win_attach = ompi_osc_monitoring_## template ##_attach, \
|
||||
.osc_win_detach = ompi_osc_monitoring_## template ##_detach, \
|
||||
.osc_free = ompi_osc_monitoring_## template ##_free, \
|
||||
\
|
||||
.osc_put = ompi_osc_monitoring_## template ##_put, \
|
||||
.osc_get = ompi_osc_monitoring_## template ##_get, \
|
||||
.osc_accumulate = ompi_osc_monitoring_## template ##_accumulate, \
|
||||
.osc_compare_and_swap = ompi_osc_monitoring_## template ##_compare_and_swap, \
|
||||
.osc_fetch_and_op = ompi_osc_monitoring_## template ##_fetch_and_op, \
|
||||
.osc_get_accumulate = ompi_osc_monitoring_## template ##_get_accumulate, \
|
||||
\
|
||||
.osc_rput = ompi_osc_monitoring_## template ##_rput, \
|
||||
.osc_rget = ompi_osc_monitoring_## template ##_rget, \
|
||||
.osc_raccumulate = ompi_osc_monitoring_## template ##_raccumulate, \
|
||||
.osc_rget_accumulate = ompi_osc_monitoring_## template ##_rget_accumulate, \
|
||||
\
|
||||
.osc_fence = ompi_osc_monitoring_## template ##_fence, \
|
||||
\
|
||||
.osc_start = ompi_osc_monitoring_## template ##_start, \
|
||||
.osc_complete = ompi_osc_monitoring_## template ##_complete, \
|
||||
.osc_post = ompi_osc_monitoring_## template ##_post, \
|
||||
.osc_wait = ompi_osc_monitoring_## template ##_wait, \
|
||||
.osc_test = ompi_osc_monitoring_## template ##_test, \
|
||||
\
|
||||
.osc_lock = ompi_osc_monitoring_## template ##_lock, \
|
||||
.osc_unlock = ompi_osc_monitoring_## template ##_unlock, \
|
||||
.osc_lock_all = ompi_osc_monitoring_## template ##_lock_all, \
|
||||
.osc_unlock_all = ompi_osc_monitoring_## template ##_unlock_all, \
|
||||
\
|
||||
.osc_sync = ompi_osc_monitoring_## template ##_sync, \
|
||||
.osc_flush = ompi_osc_monitoring_## template ##_flush, \
|
||||
.osc_flush_all = ompi_osc_monitoring_## template ##_flush_all, \
|
||||
.osc_flush_local = ompi_osc_monitoring_## template ##_flush_local, \
|
||||
.osc_flush_local_all = ompi_osc_monitoring_## template ##_flush_local_all, \
|
||||
}
|
||||
|
||||
#define OSC_MONITORING_GENERATE_TEMPLATE_MODULE(template) \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_free(ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_free(win); \
|
||||
}
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_MODULE_H */
|
||||
|
||||
63
ompi/mca/osc/monitoring/osc_monitoring_passive_target.h
Обычный файл
63
ompi/mca/osc/monitoring/osc_monitoring_passive_target.h
Обычный файл
@ -0,0 +1,63 @@
|
||||
/*
|
||||
* Copyright (c) 2016 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_PASSIVE_TARGET_H
|
||||
#define MCA_OSC_MONITORING_PASSIVE_TARGET_H
|
||||
|
||||
#include <ompi/win/win.h>
|
||||
|
||||
#define OSC_MONITORING_GENERATE_TEMPLATE_PASSIVE_TARGET(template) \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_sync (struct ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_sync(win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_flush (int target, struct ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_flush(target, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_flush_all (struct ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_flush_all(win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_flush_local (int target, struct ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_flush_local(target, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_flush_local_all (struct ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_flush_local_all(win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_lock (int lock_type, int target, int assert, ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_lock(lock_type, target, assert, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_unlock (int target, ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_unlock(target, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_lock_all (int assert, struct ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_lock_all(assert, win); \
|
||||
} \
|
||||
\
|
||||
static int ompi_osc_monitoring_## template ##_unlock_all (struct ompi_win_t *win) \
|
||||
{ \
|
||||
return OMPI_OSC_MONITORING_MODULE_VARIABLE(template).osc_unlock_all(win); \
|
||||
}
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_PASSIVE_TARGET_H */
|
||||
|
||||
79
ompi/mca/osc/monitoring/osc_monitoring_template.h
Обычный файл
79
ompi/mca/osc/monitoring/osc_monitoring_template.h
Обычный файл
@ -0,0 +1,79 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef MCA_OSC_MONITORING_TEMPLATE_H
|
||||
#define MCA_OSC_MONITORING_TEMPLATE_H
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/win/win.h>
|
||||
#include <opal/sys/atomic.h>
|
||||
#include "osc_monitoring_accumulate.h"
|
||||
#include "osc_monitoring_active_target.h"
|
||||
#include "osc_monitoring_comm.h"
|
||||
#include "osc_monitoring_dynamic.h"
|
||||
#include "osc_monitoring_module.h"
|
||||
#include "osc_monitoring_passive_target.h"
|
||||
|
||||
/* module_type correspond to the ompi_osc_## template ##_module_t type
|
||||
* comm correspond to the comm field name in ompi_osc_## template ##_module_t
|
||||
*
|
||||
* The magic used here is that for a given module type (given with the
|
||||
* {template} parameter), we generate a set of every functions defined
|
||||
* in ompi_osc_base_module_t, the ompi_osc_monitoring_module_##
|
||||
* template ##_template variable recording the original set of
|
||||
* functions, and the ompi_osc_monitoring_## template ##_template
|
||||
* variable that record the generated set of functions. When a
|
||||
* function is called from the original module, we route the call to
|
||||
* our generated function that does the monitoring, and then we call
|
||||
* the original function that had been saved in the
|
||||
* ompi_osc_monitoring_module_## template ##_template variable.
|
||||
*/
|
||||
#define OSC_MONITORING_MODULE_TEMPLATE_GENERATE(template, module_type, comm) \
|
||||
/* Generate the proper symbol for the \
|
||||
ompi_osc_monitoring_module_## template ##_template variable */ \
|
||||
OMPI_OSC_MONITORING_MODULE_GENERATE(template); \
|
||||
OMPI_OSC_MONITORING_MODULE_INIT_GENERATE(template); \
|
||||
/* Generate module specific module->comm accessor */ \
|
||||
static inline struct ompi_communicator_t* \
|
||||
ompi_osc_monitoring_## template ##_get_comm(ompi_win_t*win) \
|
||||
{ \
|
||||
return ((module_type*)win->w_osc_module)->comm; \
|
||||
} \
|
||||
/* Generate each module specific functions */ \
|
||||
OSC_MONITORING_GENERATE_TEMPLATE_ACCUMULATE(template) \
|
||||
OSC_MONITORING_GENERATE_TEMPLATE_ACTIVE_TARGET(template) \
|
||||
OSC_MONITORING_GENERATE_TEMPLATE_COMM(template) \
|
||||
OSC_MONITORING_GENERATE_TEMPLATE_DYNAMIC(template) \
|
||||
OSC_MONITORING_GENERATE_TEMPLATE_MODULE(template) \
|
||||
OSC_MONITORING_GENERATE_TEMPLATE_PASSIVE_TARGET(template) \
|
||||
/* Set the mca_osc_monitoring_## template ##_template variable */ \
|
||||
MCA_OSC_MONITORING_MODULE_TEMPLATE_GENERATE(template); \
|
||||
/* Generate template specific module initialization function */ \
|
||||
static inline void* \
|
||||
ompi_osc_monitoring_## template ##_set_template (ompi_osc_base_module_t*module) \
|
||||
{ \
|
||||
if( 1 == opal_atomic_add_64(&(OMPI_OSC_MONITORING_MODULE_INIT(template)), 1) ) { \
|
||||
/* Saves the original module functions in \
|
||||
* ompi_osc_monitoring_module_## template ##_template \
|
||||
*/ \
|
||||
memcpy(&OMPI_OSC_MONITORING_MODULE_VARIABLE(template), \
|
||||
module, sizeof(ompi_osc_base_module_t)); \
|
||||
} \
|
||||
/* Replace the original functions with our generated ones */ \
|
||||
memcpy(module, &OMPI_OSC_MONITORING_TEMPLATE_VARIABLE(template), \
|
||||
sizeof(ompi_osc_base_module_t)); \
|
||||
return module; \
|
||||
}
|
||||
|
||||
#define OSC_MONITORING_SET_TEMPLATE(template, module) \
|
||||
ompi_osc_monitoring_## template ##_set_template(module)
|
||||
|
||||
#endif /* MCA_OSC_MONITORING_TEMPLATE_H */
|
||||
|
||||
@ -11,7 +11,6 @@
|
||||
#
|
||||
|
||||
monitoring_sources = \
|
||||
pml_monitoring.c \
|
||||
pml_monitoring.h \
|
||||
pml_monitoring_comm.c \
|
||||
pml_monitoring_component.c \
|
||||
@ -32,6 +31,8 @@ mcacomponentdir = $(ompilibdir)
|
||||
mcacomponent_LTLIBRARIES = $(component_install)
|
||||
mca_pml_monitoring_la_SOURCES = $(monitoring_sources)
|
||||
mca_pml_monitoring_la_LDFLAGS = -module -avoid-version
|
||||
mca_pml_monitoring_la_LIBADD = \
|
||||
$(OMPI_TOP_BUILDDIR)/ompi/mca/common/monitoring/libmca_common_monitoring.la
|
||||
|
||||
noinst_LTLIBRARIES = $(component_noinst)
|
||||
libmca_pml_monitoring_la_SOURCES = $(monitoring_sources)
|
||||
|
||||
@ -1,258 +0,0 @@
|
||||
/*
|
||||
* Copyright (c) 2013-2016 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2015 Bull SAS. All rights reserved.
|
||||
* Copyright (c) 2016 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <pml_monitoring.h>
|
||||
#include "opal/class/opal_hash_table.h"
|
||||
|
||||
/* array for stroring monitoring data*/
|
||||
uint64_t* sent_data = NULL;
|
||||
uint64_t* messages_count = NULL;
|
||||
uint64_t* filtered_sent_data = NULL;
|
||||
uint64_t* filtered_messages_count = NULL;
|
||||
|
||||
static int init_done = 0;
|
||||
static int nbprocs = -1;
|
||||
static int my_rank = -1;
|
||||
opal_hash_table_t *translation_ht = NULL;
|
||||
|
||||
|
||||
mca_pml_monitoring_module_t mca_pml_monitoring = {
|
||||
mca_pml_monitoring_add_procs,
|
||||
mca_pml_monitoring_del_procs,
|
||||
mca_pml_monitoring_enable,
|
||||
NULL,
|
||||
mca_pml_monitoring_add_comm,
|
||||
mca_pml_monitoring_del_comm,
|
||||
mca_pml_monitoring_irecv_init,
|
||||
mca_pml_monitoring_irecv,
|
||||
mca_pml_monitoring_recv,
|
||||
mca_pml_monitoring_isend_init,
|
||||
mca_pml_monitoring_isend,
|
||||
mca_pml_monitoring_send,
|
||||
mca_pml_monitoring_iprobe,
|
||||
mca_pml_monitoring_probe,
|
||||
mca_pml_monitoring_start,
|
||||
mca_pml_monitoring_improbe,
|
||||
mca_pml_monitoring_mprobe,
|
||||
mca_pml_monitoring_imrecv,
|
||||
mca_pml_monitoring_mrecv,
|
||||
mca_pml_monitoring_dump,
|
||||
NULL,
|
||||
65535,
|
||||
INT_MAX
|
||||
};
|
||||
|
||||
/**
|
||||
* This PML monitors only the processes in the MPI_COMM_WORLD. As OMPI is now lazily
|
||||
* adding peers on the first call to add_procs we need to check how many processes
|
||||
* are in the MPI_COMM_WORLD to create the storage with the right size.
|
||||
*/
|
||||
int mca_pml_monitoring_add_procs(struct ompi_proc_t **procs,
|
||||
size_t nprocs)
|
||||
{
|
||||
opal_process_name_t tmp, wp_name;
|
||||
size_t i, peer_rank, nprocs_world;
|
||||
uint64_t key;
|
||||
|
||||
if(NULL == translation_ht) {
|
||||
translation_ht = OBJ_NEW(opal_hash_table_t);
|
||||
opal_hash_table_init(translation_ht, 2048);
|
||||
/* get my rank in the MPI_COMM_WORLD */
|
||||
my_rank = ompi_comm_rank((ompi_communicator_t*)&ompi_mpi_comm_world);
|
||||
}
|
||||
|
||||
nprocs_world = ompi_comm_size((ompi_communicator_t*)&ompi_mpi_comm_world);
|
||||
/* For all procs in the same MPI_COMM_WORLD we need to add them to the hash table */
|
||||
for( i = 0; i < nprocs; i++ ) {
|
||||
|
||||
/* Extract the peer procname from the procs array */
|
||||
if( ompi_proc_is_sentinel(procs[i]) ) {
|
||||
tmp = ompi_proc_sentinel_to_name((uintptr_t)procs[i]);
|
||||
} else {
|
||||
tmp = procs[i]->super.proc_name;
|
||||
}
|
||||
if( tmp.jobid != ompi_proc_local_proc->super.proc_name.jobid )
|
||||
continue;
|
||||
|
||||
for( peer_rank = 0; peer_rank < nprocs_world; peer_rank++ ) {
|
||||
wp_name = ompi_group_get_proc_name(((ompi_communicator_t*)&ompi_mpi_comm_world)->c_remote_group, peer_rank);
|
||||
if( 0 != opal_compare_proc( tmp, wp_name) )
|
||||
continue;
|
||||
|
||||
/* Find the rank of the peer in MPI_COMM_WORLD */
|
||||
key = *((uint64_t*)&tmp);
|
||||
/* store the rank (in COMM_WORLD) of the process
|
||||
with its name (a uniq opal ID) as key in the hash table*/
|
||||
if( OPAL_SUCCESS != opal_hash_table_set_value_uint64(translation_ht,
|
||||
key, (void*)(uintptr_t)peer_rank) ) {
|
||||
return OMPI_ERR_OUT_OF_RESOURCE; /* failed to allocate memory or growing the hash table */
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
return pml_selected_module.pml_add_procs(procs, nprocs);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pass the information down the PML stack.
|
||||
*/
|
||||
int mca_pml_monitoring_del_procs(struct ompi_proc_t **procs,
|
||||
size_t nprocs)
|
||||
{
|
||||
return pml_selected_module.pml_del_procs(procs, nprocs);
|
||||
}
|
||||
|
||||
int mca_pml_monitoring_dump(struct ompi_communicator_t* comm,
|
||||
int verbose)
|
||||
{
|
||||
return pml_selected_module.pml_dump(comm, verbose);
|
||||
}
|
||||
|
||||
|
||||
void finalize_monitoring( void )
|
||||
{
|
||||
free(filtered_sent_data);
|
||||
free(filtered_messages_count);
|
||||
free(sent_data);
|
||||
free(messages_count);
|
||||
opal_hash_table_remove_all( translation_ht );
|
||||
free(translation_ht);
|
||||
}
|
||||
|
||||
/**
|
||||
* We have delayed the initialization until the first send so that we know that
|
||||
* the MPI_COMM_WORLD (which is the only communicator we are interested on at
|
||||
* this point) is correctly initialized.
|
||||
*/
|
||||
static void initialize_monitoring( void )
|
||||
{
|
||||
nbprocs = ompi_comm_size((ompi_communicator_t*)&ompi_mpi_comm_world);
|
||||
sent_data = (uint64_t*)calloc(nbprocs, sizeof(uint64_t));
|
||||
messages_count = (uint64_t*)calloc(nbprocs, sizeof(uint64_t));
|
||||
filtered_sent_data = (uint64_t*)calloc(nbprocs, sizeof(uint64_t));
|
||||
filtered_messages_count = (uint64_t*)calloc(nbprocs, sizeof(uint64_t));
|
||||
|
||||
init_done = 1;
|
||||
}
|
||||
|
||||
void mca_pml_monitoring_reset( void )
|
||||
{
|
||||
if( !init_done ) return;
|
||||
memset(sent_data, 0, nbprocs * sizeof(uint64_t));
|
||||
memset(messages_count, 0, nbprocs * sizeof(uint64_t));
|
||||
memset(filtered_sent_data, 0, nbprocs * sizeof(uint64_t));
|
||||
memset(filtered_messages_count, 0, nbprocs * sizeof(uint64_t));
|
||||
}
|
||||
|
||||
void monitor_send_data(int world_rank, size_t data_size, int tag)
|
||||
{
|
||||
if( 0 == filter_monitoring() ) return; /* right now the monitoring is not started */
|
||||
|
||||
if ( !init_done )
|
||||
initialize_monitoring();
|
||||
|
||||
/* distinguishses positive and negative tags if requested */
|
||||
if( (tag < 0) && (1 == filter_monitoring()) ) {
|
||||
filtered_sent_data[world_rank] += data_size;
|
||||
filtered_messages_count[world_rank]++;
|
||||
} else { /* if filtered monitoring is not activated data is aggregated indifferently */
|
||||
sent_data[world_rank] += data_size;
|
||||
messages_count[world_rank]++;
|
||||
}
|
||||
}
|
||||
|
||||
int mca_pml_monitoring_get_messages_count(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int i, comm_size = ompi_comm_size (comm);
|
||||
uint64_t *values = (uint64_t*) value;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == messages_count)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = messages_count[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
int mca_pml_monitoring_get_messages_size(const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle)
|
||||
{
|
||||
ompi_communicator_t *comm = (ompi_communicator_t *) obj_handle;
|
||||
int comm_size = ompi_comm_size (comm);
|
||||
uint64_t *values = (uint64_t*) value;
|
||||
int i;
|
||||
|
||||
if(comm != &ompi_mpi_comm_world.comm || NULL == sent_data)
|
||||
return OMPI_ERROR;
|
||||
|
||||
for (i = 0 ; i < comm_size ; ++i) {
|
||||
values[i] = sent_data[i];
|
||||
}
|
||||
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static void output_monitoring( FILE *pf )
|
||||
{
|
||||
if( 0 == filter_monitoring() ) return; /* if disabled do nothing */
|
||||
|
||||
for (int i = 0 ; i < nbprocs ; i++) {
|
||||
if(sent_data[i] > 0) {
|
||||
fprintf(pf, "I\t%d\t%d\t%" PRIu64 " bytes\t%" PRIu64 " msgs sent\n",
|
||||
my_rank, i, sent_data[i], messages_count[i]);
|
||||
}
|
||||
}
|
||||
|
||||
if( 1 == filter_monitoring() ) return;
|
||||
|
||||
for (int i = 0 ; i < nbprocs ; i++) {
|
||||
if(filtered_sent_data[i] > 0) {
|
||||
fprintf(pf, "E\t%d\t%d\t%" PRIu64 " bytes\t%" PRIu64 " msgs sent\n",
|
||||
my_rank, i, filtered_sent_data[i], filtered_messages_count[i]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
Flushes the monitoring into filename
|
||||
Useful for phases (see example in test/monitoring)
|
||||
*/
|
||||
int ompi_mca_pml_monitoring_flush(char* filename)
|
||||
{
|
||||
FILE *pf = stderr;
|
||||
|
||||
if ( !init_done ) return -1;
|
||||
|
||||
if( NULL != filename )
|
||||
pf = fopen(filename, "w");
|
||||
|
||||
if(!pf)
|
||||
return -1;
|
||||
|
||||
fprintf(stderr, "Proc %d flushing monitoring to: %s\n", my_rank, filename);
|
||||
output_monitoring( pf );
|
||||
|
||||
if( NULL != filename )
|
||||
fclose(pf);
|
||||
return 0;
|
||||
}
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2015 Bull SAS. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
@ -20,14 +20,15 @@ BEGIN_C_DECLS
|
||||
#include <ompi/communicator/communicator.h>
|
||||
#include <ompi/datatype/ompi_datatype.h>
|
||||
#include <ompi/mca/pml/pml.h>
|
||||
#include <ompi/mca/pml/pml.h>
|
||||
#include <ompi/mca/pml/base/base.h>
|
||||
#include <ompi/mca/common/monitoring/common_monitoring.h>
|
||||
#include <opal/mca/base/mca_base_pvar.h>
|
||||
|
||||
typedef mca_pml_base_module_t mca_pml_monitoring_module_t;
|
||||
|
||||
extern mca_pml_base_component_t pml_selected_component;
|
||||
extern mca_pml_base_module_t pml_selected_module;
|
||||
extern mca_pml_monitoring_module_t mca_pml_monitoring;
|
||||
extern mca_pml_monitoring_module_t mca_pml_monitoring_module;
|
||||
OMPI_DECLSPEC extern mca_pml_base_component_2_0_0_t mca_pml_monitoring_component;
|
||||
|
||||
/*
|
||||
@ -38,11 +39,9 @@ extern int mca_pml_monitoring_add_comm(struct ompi_communicator_t* comm);
|
||||
|
||||
extern int mca_pml_monitoring_del_comm(struct ompi_communicator_t* comm);
|
||||
|
||||
extern int mca_pml_monitoring_add_procs(struct ompi_proc_t **procs,
|
||||
size_t nprocs);
|
||||
extern int mca_pml_monitoring_add_procs(struct ompi_proc_t **procs, size_t nprocs);
|
||||
|
||||
extern int mca_pml_monitoring_del_procs(struct ompi_proc_t **procs,
|
||||
size_t nprocs);
|
||||
extern int mca_pml_monitoring_del_procs(struct ompi_proc_t **procs, size_t nprocs);
|
||||
|
||||
extern int mca_pml_monitoring_enable(bool enable);
|
||||
|
||||
@ -138,20 +137,6 @@ extern int mca_pml_monitoring_dump(struct ompi_communicator_t* comm,
|
||||
extern int mca_pml_monitoring_start(size_t count,
|
||||
ompi_request_t** requests);
|
||||
|
||||
int mca_pml_monitoring_get_messages_count (const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
int mca_pml_monitoring_get_messages_size (const struct mca_base_pvar_t *pvar,
|
||||
void *value,
|
||||
void *obj_handle);
|
||||
|
||||
void finalize_monitoring( void );
|
||||
int filter_monitoring( void );
|
||||
void mca_pml_monitoring_reset( void );
|
||||
int ompi_mca_pml_monitoring_flush(char* filename);
|
||||
void monitor_send_data(int world_rank, size_t data_size, int tag);
|
||||
|
||||
END_C_DECLS
|
||||
|
||||
#endif /* MCA_PML_MONITORING_H */
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -11,7 +11,7 @@
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <pml_monitoring.h>
|
||||
#include "pml_monitoring.h"
|
||||
|
||||
int mca_pml_monitoring_add_comm(struct ompi_communicator_t* comm)
|
||||
{
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2016 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2015 Bull SAS. All rights reserved.
|
||||
* Copyright (c) 2015 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
@ -14,123 +14,81 @@
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <pml_monitoring.h>
|
||||
#include "pml_monitoring.h"
|
||||
#include <ompi/constants.h>
|
||||
#include <ompi/mca/pml/base/base.h>
|
||||
#include <ompi/mca/common/monitoring/common_monitoring.h>
|
||||
#include <opal/mca/base/mca_base_component_repository.h>
|
||||
|
||||
static int mca_pml_monitoring_enabled = 0;
|
||||
static int mca_pml_monitoring_active = 0;
|
||||
static int mca_pml_monitoring_current_state = 0;
|
||||
static char* mca_pml_monitoring_current_filename = NULL;
|
||||
|
||||
mca_pml_base_component_t pml_selected_component = {{0}};
|
||||
mca_pml_base_module_t pml_selected_module = {0};
|
||||
|
||||
/* Return the current status of the monitoring system 0 if off, 1 if the
|
||||
* seperation between internal tags and external tags is enabled. Any other
|
||||
* positive value if the segregation between point-to-point and collective is
|
||||
* disabled.
|
||||
mca_pml_monitoring_module_t mca_pml_monitoring_module = {
|
||||
mca_pml_monitoring_add_procs,
|
||||
mca_pml_monitoring_del_procs,
|
||||
mca_pml_monitoring_enable,
|
||||
NULL,
|
||||
mca_pml_monitoring_add_comm,
|
||||
mca_pml_monitoring_del_comm,
|
||||
mca_pml_monitoring_irecv_init,
|
||||
mca_pml_monitoring_irecv,
|
||||
mca_pml_monitoring_recv,
|
||||
mca_pml_monitoring_isend_init,
|
||||
mca_pml_monitoring_isend,
|
||||
mca_pml_monitoring_send,
|
||||
mca_pml_monitoring_iprobe,
|
||||
mca_pml_monitoring_probe,
|
||||
mca_pml_monitoring_start,
|
||||
mca_pml_monitoring_improbe,
|
||||
mca_pml_monitoring_mprobe,
|
||||
mca_pml_monitoring_imrecv,
|
||||
mca_pml_monitoring_mrecv,
|
||||
mca_pml_monitoring_dump,
|
||||
NULL,
|
||||
65535,
|
||||
INT_MAX
|
||||
};
|
||||
|
||||
/**
|
||||
* This PML monitors only the processes in the MPI_COMM_WORLD. As OMPI is now lazily
|
||||
* adding peers on the first call to add_procs we need to check how many processes
|
||||
* are in the MPI_COMM_WORLD to create the storage with the right size.
|
||||
*/
|
||||
int filter_monitoring( void )
|
||||
int mca_pml_monitoring_add_procs(struct ompi_proc_t **procs,
|
||||
size_t nprocs)
|
||||
{
|
||||
return mca_pml_monitoring_current_state;
|
||||
int ret = mca_common_monitoring_add_procs(procs, nprocs);
|
||||
if( OMPI_SUCCESS == ret )
|
||||
ret = pml_selected_module.pml_add_procs(procs, nprocs);
|
||||
return ret;
|
||||
}
|
||||
|
||||
static int
|
||||
mca_pml_monitoring_set_flush(struct mca_base_pvar_t *pvar, const void *value, void *obj)
|
||||
/**
|
||||
* Pass the information down the PML stack.
|
||||
*/
|
||||
int mca_pml_monitoring_del_procs(struct ompi_proc_t **procs,
|
||||
size_t nprocs)
|
||||
{
|
||||
if( NULL != mca_pml_monitoring_current_filename )
|
||||
free(mca_pml_monitoring_current_filename);
|
||||
if( NULL == value ) /* No more output */
|
||||
mca_pml_monitoring_current_filename = NULL;
|
||||
else {
|
||||
mca_pml_monitoring_current_filename = strdup((char*)value);
|
||||
if( NULL == mca_pml_monitoring_current_filename )
|
||||
return OMPI_ERROR;
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
return pml_selected_module.pml_del_procs(procs, nprocs);
|
||||
}
|
||||
|
||||
static int
|
||||
mca_pml_monitoring_get_flush(const struct mca_base_pvar_t *pvar, void *value, void *obj)
|
||||
int mca_pml_monitoring_dump(struct ompi_communicator_t* comm,
|
||||
int verbose)
|
||||
{
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int
|
||||
mca_pml_monitoring_notify_flush(struct mca_base_pvar_t *pvar, mca_base_pvar_event_t event,
|
||||
void *obj, int *count)
|
||||
{
|
||||
switch (event) {
|
||||
case MCA_BASE_PVAR_HANDLE_BIND:
|
||||
mca_pml_monitoring_reset();
|
||||
*count = (NULL == mca_pml_monitoring_current_filename ? 0 : strlen(mca_pml_monitoring_current_filename));
|
||||
case MCA_BASE_PVAR_HANDLE_UNBIND:
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_START:
|
||||
mca_pml_monitoring_current_state = mca_pml_monitoring_enabled;
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_STOP:
|
||||
if( 0 == ompi_mca_pml_monitoring_flush(mca_pml_monitoring_current_filename) )
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
return OMPI_ERROR;
|
||||
}
|
||||
|
||||
static int
|
||||
mca_pml_monitoring_messages_notify(mca_base_pvar_t *pvar,
|
||||
mca_base_pvar_event_t event,
|
||||
void *obj_handle,
|
||||
int *count)
|
||||
{
|
||||
switch (event) {
|
||||
case MCA_BASE_PVAR_HANDLE_BIND:
|
||||
/* Return the size of the communicator as the number of values */
|
||||
*count = ompi_comm_size ((ompi_communicator_t *) obj_handle);
|
||||
case MCA_BASE_PVAR_HANDLE_UNBIND:
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_START:
|
||||
mca_pml_monitoring_current_state = mca_pml_monitoring_enabled;
|
||||
return OMPI_SUCCESS;
|
||||
case MCA_BASE_PVAR_HANDLE_STOP:
|
||||
mca_pml_monitoring_current_state = 0;
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
return OMPI_ERROR;
|
||||
return pml_selected_module.pml_dump(comm, verbose);
|
||||
}
|
||||
|
||||
int mca_pml_monitoring_enable(bool enable)
|
||||
{
|
||||
/* If we reach this point we were succesful at hijacking the interface of
|
||||
* the real PML, and we are now correctly interleaved between the upper
|
||||
* layer and the real PML.
|
||||
*/
|
||||
(void)mca_base_pvar_register("ompi", "pml", "monitoring", "flush", "Flush the monitoring information"
|
||||
"in the provided file", OPAL_INFO_LVL_1, MCA_BASE_PVAR_CLASS_GENERIC,
|
||||
MCA_BASE_VAR_TYPE_STRING, NULL, MPI_T_BIND_NO_OBJECT,
|
||||
0,
|
||||
mca_pml_monitoring_get_flush, mca_pml_monitoring_set_flush,
|
||||
mca_pml_monitoring_notify_flush, &mca_pml_monitoring_component);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "pml", "monitoring", "messages_count", "Number of messages "
|
||||
"sent to each peer in a communicator", OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_BASE_VAR_TYPE_UNSIGNED_LONG, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_pml_monitoring_get_messages_count, NULL, mca_pml_monitoring_messages_notify, NULL);
|
||||
|
||||
(void)mca_base_pvar_register("ompi", "pml", "monitoring", "messages_size", "Size of messages "
|
||||
"sent to each peer in a communicator", OPAL_INFO_LVL_4, MPI_T_PVAR_CLASS_SIZE,
|
||||
MCA_BASE_VAR_TYPE_UNSIGNED_LONG, NULL, MPI_T_BIND_MPI_COMM,
|
||||
MCA_BASE_PVAR_FLAG_READONLY,
|
||||
mca_pml_monitoring_get_messages_size, NULL, mca_pml_monitoring_messages_notify, NULL);
|
||||
|
||||
return pml_selected_module.pml_enable(enable);
|
||||
}
|
||||
|
||||
static int mca_pml_monitoring_component_open(void)
|
||||
{
|
||||
if( mca_pml_monitoring_enabled ) {
|
||||
/* CF: What if we are the only PML available ?? */
|
||||
if( mca_common_monitoring_enabled ) {
|
||||
opal_pointer_array_add(&mca_pml_base_pml,
|
||||
strdup(mca_pml_monitoring_component.pmlm_version.mca_component_name));
|
||||
}
|
||||
@ -139,22 +97,15 @@ static int mca_pml_monitoring_component_open(void)
|
||||
|
||||
static int mca_pml_monitoring_component_close(void)
|
||||
{
|
||||
if( NULL != mca_pml_monitoring_current_filename ) {
|
||||
free(mca_pml_monitoring_current_filename);
|
||||
mca_pml_monitoring_current_filename = NULL;
|
||||
}
|
||||
if( !mca_pml_monitoring_enabled )
|
||||
return OMPI_SUCCESS;
|
||||
if( !mca_common_monitoring_enabled ) return OMPI_SUCCESS;
|
||||
|
||||
/**
|
||||
* If this component is already active, then we are currently monitoring the execution
|
||||
* and this close if the one from MPI_Finalize. Do the clean up and release the extra
|
||||
* reference on ourselves.
|
||||
* If this component is already active, then we are currently monitoring
|
||||
* the execution and this call to close if the one from MPI_Finalize.
|
||||
* Clean up and release the extra reference on ourselves.
|
||||
*/
|
||||
if( mca_pml_monitoring_active ) { /* Already active, turn off */
|
||||
pml_selected_component.pmlm_version.mca_close_component();
|
||||
memset(&pml_selected_component, 0, sizeof(mca_pml_base_component_t));
|
||||
memset(&pml_selected_module, 0, sizeof(mca_pml_base_module_t));
|
||||
mca_base_component_repository_release((mca_base_component_t*)&mca_pml_monitoring_component);
|
||||
mca_pml_monitoring_active = 0;
|
||||
return OMPI_SUCCESS;
|
||||
@ -175,12 +126,13 @@ static int mca_pml_monitoring_component_close(void)
|
||||
pml_selected_module = mca_pml;
|
||||
/* Install our interception layer */
|
||||
mca_pml_base_selected_component = mca_pml_monitoring_component;
|
||||
mca_pml = mca_pml_monitoring;
|
||||
/* Restore some of the original valued: progress, flags, tags and context id */
|
||||
mca_pml = mca_pml_monitoring_module;
|
||||
/* Restore some of the original values: progress, flags, tags and context id */
|
||||
mca_pml.pml_progress = pml_selected_module.pml_progress;
|
||||
mca_pml.pml_max_contextid = pml_selected_module.pml_max_contextid;
|
||||
mca_pml.pml_max_tag = pml_selected_module.pml_max_tag;
|
||||
mca_pml.pml_flags = pml_selected_module.pml_flags;
|
||||
/* Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to ensure the hashtable is properly initialized */
|
||||
mca_pml.pml_flags = pml_selected_module.pml_flags | MCA_PML_BASE_FLAG_REQUIRE_WORLD;
|
||||
|
||||
mca_pml_monitoring_active = 1;
|
||||
|
||||
@ -192,44 +144,36 @@ mca_pml_monitoring_component_init(int* priority,
|
||||
bool enable_progress_threads,
|
||||
bool enable_mpi_threads)
|
||||
{
|
||||
if( mca_pml_monitoring_enabled ) {
|
||||
mca_common_monitoring_init();
|
||||
if( mca_common_monitoring_enabled ) {
|
||||
*priority = 0; /* I'm up but don't select me */
|
||||
return &mca_pml_monitoring;
|
||||
return &mca_pml_monitoring_module;
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static int mca_pml_monitoring_component_finish(void)
|
||||
{
|
||||
if( mca_pml_monitoring_enabled && mca_pml_monitoring_active ) {
|
||||
if( mca_common_monitoring_enabled && mca_pml_monitoring_active ) {
|
||||
/* Free internal data structure */
|
||||
finalize_monitoring();
|
||||
/* Call the original PML and then close */
|
||||
mca_pml_monitoring_active = 0;
|
||||
mca_pml_monitoring_enabled = 0;
|
||||
mca_common_monitoring_finalize();
|
||||
/* Restore the original PML */
|
||||
mca_pml_base_selected_component = pml_selected_component;
|
||||
mca_pml = pml_selected_module;
|
||||
/* Redirect the close call to the original PML */
|
||||
pml_selected_component.pmlm_finalize();
|
||||
/**
|
||||
* We should never release the last ref on the current component or face forever punishement.
|
||||
* We should never release the last ref on the current
|
||||
* component or face forever punishement.
|
||||
*/
|
||||
/* mca_base_component_repository_release(&mca_pml_monitoring_component.pmlm_version); */
|
||||
/* mca_base_component_repository_release(&mca_common_monitoring_component.pmlm_version); */
|
||||
}
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
static int mca_pml_monitoring_component_register(void)
|
||||
{
|
||||
(void)mca_base_component_var_register(&mca_pml_monitoring_component.pmlm_version, "enable",
|
||||
"Enable the monitoring at the PML level. A value of 0 will disable the monitoring (default). "
|
||||
"A value of 1 will aggregate all monitoring information (point-to-point and collective). "
|
||||
"Any other value will enable filtered monitoring",
|
||||
MCA_BASE_VAR_TYPE_INT, NULL, 0, 0,
|
||||
OPAL_INFO_LVL_4,
|
||||
MCA_BASE_VAR_SCOPE_READONLY, &mca_pml_monitoring_enabled);
|
||||
|
||||
mca_common_monitoring_register(&mca_pml_monitoring_component);
|
||||
return OMPI_SUCCESS;
|
||||
}
|
||||
|
||||
@ -242,9 +186,7 @@ mca_pml_base_component_2_0_0_t mca_pml_monitoring_component = {
|
||||
MCA_PML_BASE_VERSION_2_0_0,
|
||||
|
||||
.mca_component_name = "monitoring", /* MCA component name */
|
||||
.mca_component_major_version = OMPI_MAJOR_VERSION, /* MCA component major version */
|
||||
.mca_component_minor_version = OMPI_MINOR_VERSION, /* MCA component minor version */
|
||||
.mca_component_release_version = OMPI_RELEASE_VERSION, /* MCA component release version */
|
||||
MCA_MONITORING_MAKE_VERSION,
|
||||
.mca_open_component = mca_pml_monitoring_component_open, /* component open */
|
||||
.mca_close_component = mca_pml_monitoring_component_close, /* component close */
|
||||
.mca_register_component_params = mca_pml_monitoring_component_register
|
||||
@ -256,6 +198,5 @@ mca_pml_base_component_2_0_0_t mca_pml_monitoring_component = {
|
||||
|
||||
.pmlm_init = mca_pml_monitoring_component_init, /* component init */
|
||||
.pmlm_finalize = mca_pml_monitoring_component_finish /* component finalize */
|
||||
|
||||
};
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -11,7 +11,7 @@
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <pml_monitoring.h>
|
||||
#include "pml_monitoring.h"
|
||||
|
||||
|
||||
/* EJ: nothing to do here */
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -11,7 +11,7 @@
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <pml_monitoring.h>
|
||||
#include "pml_monitoring.h"
|
||||
|
||||
|
||||
/* EJ: loging is done on the sender. Nothing to do here */
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -11,9 +11,7 @@
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <pml_monitoring.h>
|
||||
|
||||
extern opal_hash_table_t *translation_ht;
|
||||
#include "pml_monitoring.h"
|
||||
|
||||
int mca_pml_monitoring_isend_init(const void *buf,
|
||||
size_t count,
|
||||
@ -37,22 +35,16 @@ int mca_pml_monitoring_isend(const void *buf,
|
||||
struct ompi_communicator_t* comm,
|
||||
struct ompi_request_t **request)
|
||||
{
|
||||
|
||||
/* find the processor of teh destination */
|
||||
ompi_proc_t *proc = ompi_group_get_proc_ptr(comm->c_remote_group, dst, true);
|
||||
int world_rank;
|
||||
|
||||
/* find its name*/
|
||||
uint64_t key = *((uint64_t*)&(proc->super.proc_name));
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
* Lookup its name in the rank hastable to get its MPI_COMM_WORLD rank
|
||||
*/
|
||||
if(OPAL_SUCCESS == opal_hash_table_get_value_uint64(translation_ht, key, (void *)&world_rank)) {
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(dst, comm, &world_rank)) {
|
||||
size_t type_size, data_size;
|
||||
ompi_datatype_type_size(datatype, &type_size);
|
||||
data_size = count*type_size;
|
||||
monitor_send_data(world_rank, data_size, tag);
|
||||
mca_common_monitoring_record_pml(world_rank, data_size, tag);
|
||||
}
|
||||
|
||||
return pml_selected_module.pml_isend(buf, count, datatype,
|
||||
@ -67,19 +59,15 @@ int mca_pml_monitoring_send(const void *buf,
|
||||
mca_pml_base_send_mode_t mode,
|
||||
struct ompi_communicator_t* comm)
|
||||
{
|
||||
ompi_proc_t *proc = ompi_group_get_proc_ptr(comm->c_remote_group, dst, true);
|
||||
int world_rank;
|
||||
uint64_t key = *((uint64_t*) &(proc->super.proc_name));
|
||||
|
||||
/* Are we sending to a peer from my own MPI_COMM_WORLD? */
|
||||
if(OPAL_SUCCESS == opal_hash_table_get_value_uint64(translation_ht, key, (void *)&world_rank)) {
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(dst, comm, &world_rank)) {
|
||||
size_t type_size, data_size;
|
||||
ompi_datatype_type_size(datatype, &type_size);
|
||||
data_size = count*type_size;
|
||||
monitor_send_data(world_rank, data_size, tag);
|
||||
mca_common_monitoring_record_pml(world_rank, data_size, tag);
|
||||
}
|
||||
|
||||
return pml_selected_module.pml_send(buf, count, datatype,
|
||||
dst, tag, mode, comm);
|
||||
}
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -11,12 +11,9 @@
|
||||
*/
|
||||
|
||||
#include <ompi_config.h>
|
||||
#include <pml_monitoring.h>
|
||||
#include <opal/class/opal_hash_table.h>
|
||||
#include "pml_monitoring.h"
|
||||
#include <ompi/mca/pml/base/pml_base_request.h>
|
||||
|
||||
extern opal_hash_table_t *translation_ht;
|
||||
|
||||
/* manage persistant requests*/
|
||||
int mca_pml_monitoring_start(size_t count,
|
||||
ompi_request_t** requests)
|
||||
@ -25,7 +22,6 @@ int mca_pml_monitoring_start(size_t count,
|
||||
|
||||
for( i = 0; i < count; i++ ) {
|
||||
mca_pml_base_request_t *pml_request = (mca_pml_base_request_t*)requests[i];
|
||||
ompi_proc_t *proc;
|
||||
int world_rank;
|
||||
|
||||
if(NULL == pml_request) {
|
||||
@ -38,18 +34,15 @@ int mca_pml_monitoring_start(size_t count,
|
||||
continue;
|
||||
}
|
||||
|
||||
proc = ompi_group_get_proc_ptr(pml_request->req_comm->c_remote_group, pml_request->req_peer, true);
|
||||
uint64_t key = *((uint64_t*) &(proc->super.proc_name));
|
||||
|
||||
|
||||
/**
|
||||
* If this fails the destination is not part of my MPI_COM_WORLD
|
||||
*/
|
||||
if(OPAL_SUCCESS == opal_hash_table_get_value_uint64(translation_ht, key, (void *)&world_rank)) {
|
||||
if(OPAL_SUCCESS == mca_common_monitoring_get_world_rank(pml_request->req_peer,
|
||||
pml_request->req_comm, &world_rank)) {
|
||||
size_t type_size, data_size;
|
||||
ompi_datatype_type_size(pml_request->req_datatype, &type_size);
|
||||
data_size = pml_request->req_count * type_size;
|
||||
monitor_send_data(world_rank, data_size, 1);
|
||||
mca_common_monitoring_record_pml(world_rank, data_size, 1);
|
||||
}
|
||||
}
|
||||
return pml_selected_module.pml_start(count, requests);
|
||||
|
||||
@ -719,6 +719,8 @@ int mca_base_pvar_handle_write_value (mca_base_pvar_handle_t *handle, const void
|
||||
}
|
||||
|
||||
memmove (handle->current_value, value, handle->count * var_type_sizes[handle->pvar->type]);
|
||||
/* read the value directly from the variable. */
|
||||
ret = handle->pvar->set_value (handle->pvar, value, handle->obj_handle);
|
||||
|
||||
return OPAL_SUCCESS;
|
||||
}
|
||||
|
||||
@ -1,12 +1,12 @@
|
||||
#
|
||||
# Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
# Copyright (c) 2013-2017 The University of Tennessee and The University
|
||||
# of Tennessee Research Foundation. All rights
|
||||
# reserved.
|
||||
# Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
# Copyright (c) 2015 Research Organization for Information Science
|
||||
# Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
# Copyright (c) 2015-2017 Research Organization for Information Science
|
||||
# and Technology (RIST). All rights reserved.
|
||||
# Copyright (c) 2016 IBM Corporation. All rights reserved.
|
||||
# Copyright (c) 2016 Cisco Systems, Inc. All rights reserved.
|
||||
# Copyright (c) 2016 Cisco Systems, Inc. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
@ -14,15 +14,37 @@
|
||||
# $HEADER$
|
||||
#
|
||||
|
||||
EXTRA_DIST = profile2mat.pl aggregate_profile.pl
|
||||
|
||||
# This test requires multiple processes to run. Don't run it as part
|
||||
# of 'make check'
|
||||
if PROJECT_OMPI
|
||||
noinst_PROGRAMS = monitoring_test
|
||||
noinst_PROGRAMS = monitoring_test test_pvar_access test_overhead check_monitoring example_reduce_count
|
||||
monitoring_test_SOURCES = monitoring_test.c
|
||||
monitoring_test_LDFLAGS = $(OMPI_PKG_CONFIG_LDFLAGS)
|
||||
monitoring_test_LDADD = \
|
||||
$(top_builddir)/ompi/lib@OMPI_LIBMPI_NAME@.la \
|
||||
$(top_builddir)/opal/lib@OPAL_LIB_PREFIX@open-pal.la
|
||||
test_pvar_access_SOURCES = test_pvar_access.c
|
||||
test_pvar_access_LDFLAGS = $(OMPI_PKG_CONFIG_LDFLAGS)
|
||||
test_pvar_access_LDADD = \
|
||||
$(top_builddir)/ompi/lib@OMPI_LIBMPI_NAME@.la \
|
||||
$(top_builddir)/opal/lib@OPAL_LIB_PREFIX@open-pal.la
|
||||
test_overhead_SOURCES = test_overhead.c
|
||||
test_overhead_LDFLAGS = $(OMPI_PKG_CONFIG_LDFLAGS)
|
||||
test_overhead_LDADD = \
|
||||
$(top_builddir)/ompi/lib@OMPI_LIBMPI_NAME@.la \
|
||||
$(top_builddir)/opal/lib@OPAL_LIB_PREFIX@open-pal.la
|
||||
check_monitoring_SOURCES = check_monitoring.c
|
||||
check_monitoring_LDFLAGS = $(OMPI_PKG_CONFIG_LDFLAGS)
|
||||
check_monitoring_LDADD = \
|
||||
$(top_builddir)/ompi/lib@OMPI_LIBMPI_NAME@.la \
|
||||
$(top_builddir)/opal/lib@OPAL_LIB_PREFIX@open-pal.la
|
||||
example_reduce_count_SOURCES = example_reduce_count.c
|
||||
example_reduce_count_LDFLAGS = $(OMPI_PKG_CONFIG_LDFLAGS)
|
||||
example_reduce_count_LDADD = \
|
||||
$(top_builddir)/ompi/lib@OMPI_LIBMPI_NAME@.la \
|
||||
$(top_builddir)/opal/lib@OPAL_LIB_PREFIX@open-pal.la
|
||||
|
||||
if MCA_BUILD_ompi_pml_monitoring_DSO
|
||||
lib_LTLIBRARIES = ompi_monitoring_prof.la
|
||||
@ -34,4 +56,11 @@ if MCA_BUILD_ompi_pml_monitoring_DSO
|
||||
$(top_builddir)/opal/lib@OPAL_LIB_PREFIX@open-pal.la
|
||||
endif # MCA_BUILD_ompi_pml_monitoring_DSO
|
||||
|
||||
if OPAL_INSTALL_BINARIES
|
||||
bin_SCRIPTS = profile2mat.pl aggregate_profile.pl
|
||||
endif # OPAL_INSTALL_BINARIES
|
||||
|
||||
endif # PROJECT_OMPI
|
||||
|
||||
distclean:
|
||||
rm -rf *.dSYM .deps .libs *.la *.lo monitoring_test test_pvar_access test_overhead check_monitoring example_reduce_count prof *.log *.o *.trs Makefile
|
||||
|
||||
@ -28,7 +28,7 @@
|
||||
# ensure that this script as the executable right: chmod +x ...
|
||||
#
|
||||
|
||||
die "$0 <name of the profile>\n\tProfile files should be of the form \"name_phaseid_processesid.prof\"\n\tFor instance if you saved the monitoring into phase_0_0.prof, phase_0_1.prof, ..., phase_1_0.prof etc you should call: $0 phase\n" if ($#ARGV!=0);
|
||||
die "$0 <name of the profile>\n\tProfile files should be of the form \"name_phaseid_processesid.prof\"\n\tFor instance if you saved the monitoring into phase_0.0.prof, phase_0.1.prof, ..., phase_1.0.prof etc you should call: $0 phase\n" if ($#ARGV!=0);
|
||||
|
||||
$name = $ARGV[0];
|
||||
|
||||
@ -39,7 +39,7 @@ $name = $ARGV[0];
|
||||
|
||||
# Detect the different phases
|
||||
foreach $file (@files) {
|
||||
($id)=($file =~ m/$name\_(\d+)_\d+/);
|
||||
($id)=($file =~ m/$name\_(\d+)\.\d+/);
|
||||
$phaseid{$id} = 1 if ($id);
|
||||
}
|
||||
|
||||
|
||||
516
test/monitoring/check_monitoring.c
Обычный файл
516
test/monitoring/check_monitoring.c
Обычный файл
@ -0,0 +1,516 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2017 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
/*
|
||||
Check the well working of the monitoring component for Open-MPI.
|
||||
|
||||
To be run as:
|
||||
|
||||
mpirun -np 4 --mca pml_monitoring_enable 2 ./check_monitoring
|
||||
*/
|
||||
|
||||
#include <mpi.h>
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
|
||||
#define PVAR_GENERATE_VARIABLES(pvar_prefix, pvar_name, pvar_class) \
|
||||
/* Variables */ \
|
||||
static MPI_T_pvar_handle pvar_prefix ## _handle; \
|
||||
static const char pvar_prefix ## _pvar_name[] = pvar_name; \
|
||||
static int pvar_prefix ## _pvar_idx; \
|
||||
/* Functions */ \
|
||||
static inline int pvar_prefix ## _start(MPI_T_pvar_session session) \
|
||||
{ \
|
||||
int MPIT_result; \
|
||||
MPIT_result = MPI_T_pvar_start(session, pvar_prefix ## _handle); \
|
||||
if( MPI_SUCCESS != MPIT_result ) { \
|
||||
fprintf(stderr, "Failed to start handle on \"%s\" pvar, check that you have " \
|
||||
"enabled the monitoring component.\n", pvar_prefix ## _pvar_name); \
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result); \
|
||||
} \
|
||||
return MPIT_result; \
|
||||
} \
|
||||
static inline int pvar_prefix ## _init(MPI_T_pvar_session session) \
|
||||
{ \
|
||||
int MPIT_result; \
|
||||
/* Get index */ \
|
||||
MPIT_result = MPI_T_pvar_get_index(pvar_prefix ## _pvar_name, \
|
||||
pvar_class, \
|
||||
&(pvar_prefix ## _pvar_idx)); \
|
||||
if( MPI_SUCCESS != MPIT_result ) { \
|
||||
fprintf(stderr, "Cannot find monitoring MPI_Tool \"%s\" pvar, check that you have " \
|
||||
"enabled the monitoring component.\n", pvar_prefix ## _pvar_name); \
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result); \
|
||||
return MPIT_result; \
|
||||
} \
|
||||
/* Allocate handle */ \
|
||||
/* Allocating a new PVAR in a session will reset the counters */ \
|
||||
int count; \
|
||||
MPIT_result = MPI_T_pvar_handle_alloc(session, pvar_prefix ## _pvar_idx, \
|
||||
MPI_COMM_WORLD, &(pvar_prefix ## _handle), \
|
||||
&count); \
|
||||
if( MPI_SUCCESS != MPIT_result ) { \
|
||||
fprintf(stderr, "Failed to allocate handle on \"%s\" pvar, check that you have " \
|
||||
"enabled the monitoring component.\n", pvar_prefix ## _pvar_name); \
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result); \
|
||||
return MPIT_result; \
|
||||
} \
|
||||
/* Start PVAR */ \
|
||||
return pvar_prefix ## _start(session); \
|
||||
} \
|
||||
static inline int pvar_prefix ## _stop(MPI_T_pvar_session session) \
|
||||
{ \
|
||||
int MPIT_result; \
|
||||
MPIT_result = MPI_T_pvar_stop(session, pvar_prefix ## _handle); \
|
||||
if( MPI_SUCCESS != MPIT_result ) { \
|
||||
fprintf(stderr, "Failed to stop handle on \"%s\" pvar, check that you have " \
|
||||
"enabled the monitoring component.\n", pvar_prefix ## _pvar_name); \
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result); \
|
||||
} \
|
||||
return MPIT_result; \
|
||||
} \
|
||||
static inline int pvar_prefix ## _finalize(MPI_T_pvar_session session) \
|
||||
{ \
|
||||
int MPIT_result; \
|
||||
/* Stop PVAR */ \
|
||||
MPIT_result = pvar_prefix ## _stop(session); \
|
||||
/* Free handle */ \
|
||||
MPIT_result = MPI_T_pvar_handle_free(session, &(pvar_prefix ## _handle)); \
|
||||
if( MPI_SUCCESS != MPIT_result ) { \
|
||||
fprintf(stderr, "Failed to allocate handle on \"%s\" pvar, check that you have " \
|
||||
"enabled the monitoring component.\n", pvar_prefix ## _pvar_name); \
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result); \
|
||||
return MPIT_result; \
|
||||
} \
|
||||
return MPIT_result; \
|
||||
} \
|
||||
static inline int pvar_prefix ## _read(MPI_T_pvar_session session, void*values) \
|
||||
{ \
|
||||
int MPIT_result; \
|
||||
/* Stop pvar */ \
|
||||
MPIT_result = pvar_prefix ## _stop(session); \
|
||||
/* Read values */ \
|
||||
MPIT_result = MPI_T_pvar_read(session, pvar_prefix ## _handle, values); \
|
||||
if( MPI_SUCCESS != MPIT_result ) { \
|
||||
fprintf(stderr, "Failed to read handle on \"%s\" pvar, check that you have " \
|
||||
"enabled the monitoring component.\n", pvar_prefix ## _pvar_name); \
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result); \
|
||||
} \
|
||||
/* Start and return */ \
|
||||
return pvar_prefix ## _start(session); \
|
||||
}
|
||||
|
||||
#define GENERATE_CS(prefix, pvar_name_prefix, pvar_class_c, pvar_class_s) \
|
||||
PVAR_GENERATE_VARIABLES(prefix ## _count, pvar_name_prefix "_count", pvar_class_c) \
|
||||
PVAR_GENERATE_VARIABLES(prefix ## _size, pvar_name_prefix "_size", pvar_class_s) \
|
||||
static inline int pvar_ ## prefix ## _init(MPI_T_pvar_session session) \
|
||||
{ \
|
||||
prefix ## _count_init(session); \
|
||||
return prefix ## _size_init(session); \
|
||||
} \
|
||||
static inline int pvar_ ## prefix ## _finalize(MPI_T_pvar_session session) \
|
||||
{ \
|
||||
prefix ## _count_finalize(session); \
|
||||
return prefix ## _size_finalize(session); \
|
||||
} \
|
||||
static inline void pvar_ ## prefix ## _read(MPI_T_pvar_session session, \
|
||||
size_t*cvalues, size_t*svalues) \
|
||||
{ \
|
||||
/* Read count values */ \
|
||||
prefix ## _count_read(session, cvalues); \
|
||||
/* Read size values */ \
|
||||
prefix ## _size_read(session, svalues); \
|
||||
}
|
||||
|
||||
GENERATE_CS(pml, "pml_monitoring_messages", MPI_T_PVAR_CLASS_SIZE, MPI_T_PVAR_CLASS_SIZE)
|
||||
GENERATE_CS(osc_s, "osc_monitoring_messages_sent", MPI_T_PVAR_CLASS_SIZE, MPI_T_PVAR_CLASS_SIZE)
|
||||
GENERATE_CS(osc_r, "osc_monitoring_messages_recv", MPI_T_PVAR_CLASS_SIZE, MPI_T_PVAR_CLASS_SIZE)
|
||||
GENERATE_CS(coll, "coll_monitoring_messages", MPI_T_PVAR_CLASS_SIZE, MPI_T_PVAR_CLASS_SIZE)
|
||||
GENERATE_CS(o2a, "coll_monitoring_o2a", MPI_T_PVAR_CLASS_COUNTER, MPI_T_PVAR_CLASS_AGGREGATE)
|
||||
GENERATE_CS(a2o, "coll_monitoring_a2o", MPI_T_PVAR_CLASS_COUNTER, MPI_T_PVAR_CLASS_AGGREGATE)
|
||||
GENERATE_CS(a2a, "coll_monitoring_a2a", MPI_T_PVAR_CLASS_COUNTER, MPI_T_PVAR_CLASS_AGGREGATE)
|
||||
|
||||
static size_t *old_cvalues, *old_svalues;
|
||||
|
||||
static inline void pvar_all_init(MPI_T_pvar_session*session, int world_size)
|
||||
{
|
||||
int MPIT_result, provided;
|
||||
MPIT_result = MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
fprintf(stderr, "Failed to initialiaze MPI_Tools sub-system.\n");
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_session_create(session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("Failed to create a session for PVARs.\n");
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
old_cvalues = malloc(2 * world_size * sizeof(size_t));
|
||||
old_svalues = old_cvalues + world_size;
|
||||
pvar_pml_init(*session);
|
||||
pvar_osc_s_init(*session);
|
||||
pvar_osc_r_init(*session);
|
||||
pvar_coll_init(*session);
|
||||
pvar_o2a_init(*session);
|
||||
pvar_a2o_init(*session);
|
||||
pvar_a2a_init(*session);
|
||||
}
|
||||
|
||||
static inline void pvar_all_finalize(MPI_T_pvar_session*session)
|
||||
{
|
||||
int MPIT_result;
|
||||
pvar_pml_finalize(*session);
|
||||
pvar_osc_s_finalize(*session);
|
||||
pvar_osc_r_finalize(*session);
|
||||
pvar_coll_finalize(*session);
|
||||
pvar_o2a_finalize(*session);
|
||||
pvar_a2o_finalize(*session);
|
||||
pvar_a2a_finalize(*session);
|
||||
free(old_cvalues);
|
||||
MPIT_result = MPI_T_pvar_session_free(session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("Failed to close a session for PVARs.\n");
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
(void)MPI_T_finalize();
|
||||
}
|
||||
|
||||
static inline int pvar_pml_check(MPI_T_pvar_session session, int world_size, int world_rank)
|
||||
{
|
||||
int i, ret = MPI_SUCCESS;
|
||||
size_t *cvalues, *svalues;
|
||||
cvalues = malloc(2 * world_size * sizeof(size_t));
|
||||
svalues = cvalues + world_size;
|
||||
/* Get values */
|
||||
pvar_pml_read(session, cvalues, svalues);
|
||||
for( i = 0; i < world_size && MPI_SUCCESS == ret; ++i ) {
|
||||
/* Check count values */
|
||||
if( i == world_rank && (cvalues[i] - old_cvalues[i]) != (size_t) 0 ) {
|
||||
fprintf(stderr, "Error in %s: count_values[%d]=%zu, and should be equal to %zu.\n",
|
||||
__func__, i, cvalues[i] - old_cvalues[i], (size_t) 0);
|
||||
ret = -1;
|
||||
} else if ( i != world_rank && (cvalues[i] - old_cvalues[i]) < (size_t) world_size ) {
|
||||
fprintf(stderr, "Error in %s: count_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, cvalues[i] - old_cvalues[i], (size_t) world_size);
|
||||
ret = -1;
|
||||
}
|
||||
/* Check size values */
|
||||
if( i == world_rank && (svalues[i] - old_svalues[i]) != (size_t) 0 ) {
|
||||
fprintf(stderr, "Error in %s: size_values[%d]=%zu, and should be equal to %zu.\n",
|
||||
__func__, i, svalues[i] - old_svalues[i], (size_t) 0);
|
||||
ret = -1;
|
||||
} else if ( i != world_rank && (svalues[i] - old_svalues[i]) < (size_t) (world_size * 13 * sizeof(char)) ) {
|
||||
fprintf(stderr, "Error in %s: size_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, svalues[i] - old_svalues[i], (size_t) (world_size * 13 * sizeof(char)));
|
||||
ret = -1;
|
||||
}
|
||||
}
|
||||
if( MPI_SUCCESS == ret ) {
|
||||
fprintf(stdout, "Check PML...[ OK ]\n");
|
||||
} else {
|
||||
fprintf(stdout, "Check PML...[FAIL]\n");
|
||||
}
|
||||
/* Keep old PML values */
|
||||
memcpy(old_cvalues, cvalues, 2 * world_size * sizeof(size_t));
|
||||
/* Free arrays */
|
||||
free(cvalues);
|
||||
return ret;
|
||||
}
|
||||
|
||||
static inline int pvar_osc_check(MPI_T_pvar_session session, int world_size, int world_rank)
|
||||
{
|
||||
int i, ret = MPI_SUCCESS;
|
||||
size_t *cvalues, *svalues;
|
||||
cvalues = malloc(2 * world_size * sizeof(size_t));
|
||||
svalues = cvalues + world_size;
|
||||
/* Get OSC values */
|
||||
memset(cvalues, 0, 2 * world_size * sizeof(size_t));
|
||||
/* Check OSC sent values */
|
||||
pvar_osc_s_read(session, cvalues, svalues);
|
||||
for( i = 0; i < world_size && MPI_SUCCESS == ret; ++i ) {
|
||||
/* Check count values */
|
||||
if( cvalues[i] < (size_t) world_size ) {
|
||||
fprintf(stderr, "Error in %s: count_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, cvalues[i], (size_t) world_size);
|
||||
ret = -1;
|
||||
}
|
||||
/* Check size values */
|
||||
if( svalues[i] < (size_t) (world_size * 13 * sizeof(char)) ) {
|
||||
fprintf(stderr, "Error in %s: size_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, svalues[i], (size_t) (world_size * 13 * sizeof(char)));
|
||||
ret = -1;
|
||||
}
|
||||
}
|
||||
/* Check OSC received values */
|
||||
pvar_osc_r_read(session, cvalues, svalues);
|
||||
for( i = 0; i < world_size && MPI_SUCCESS == ret; ++i ) {
|
||||
/* Check count values */
|
||||
if( cvalues[i] < (size_t) world_size ) {
|
||||
fprintf(stderr, "Error in %s: count_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, cvalues[i], (size_t) world_size);
|
||||
ret = -1;
|
||||
}
|
||||
/* Check size values */
|
||||
if( svalues[i] < (size_t) (world_size * 13 * sizeof(char)) ) {
|
||||
fprintf(stderr, "Error in %s: size_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, svalues[i], (size_t) (world_size * 13 * sizeof(char)));
|
||||
ret = -1;
|
||||
}
|
||||
}
|
||||
if( MPI_SUCCESS == ret ) {
|
||||
fprintf(stdout, "Check OSC...[ OK ]\n");
|
||||
} else {
|
||||
fprintf(stdout, "Check OSC...[FAIL]\n");
|
||||
}
|
||||
/* Keep old PML values */
|
||||
memcpy(old_cvalues, cvalues, 2 * world_size * sizeof(size_t));
|
||||
/* Free arrays */
|
||||
free(cvalues);
|
||||
return ret;
|
||||
}
|
||||
|
||||
static inline int pvar_coll_check(MPI_T_pvar_session session, int world_size, int world_rank) {
|
||||
int i, ret = MPI_SUCCESS;
|
||||
size_t count, size;
|
||||
size_t *cvalues, *svalues;
|
||||
cvalues = malloc(2 * world_size * sizeof(size_t));
|
||||
svalues = cvalues + world_size;
|
||||
/* Get COLL values */
|
||||
pvar_coll_read(session, cvalues, svalues);
|
||||
for( i = 0; i < world_size && MPI_SUCCESS == ret; ++i ) {
|
||||
/* Check count values */
|
||||
if( i == world_rank && cvalues[i] != (size_t) 0 ) {
|
||||
fprintf(stderr, "Error in %s: count_values[%d]=%zu, and should be equal to %zu.\n",
|
||||
__func__, i, cvalues[i], (size_t) 0);
|
||||
ret = -1;
|
||||
} else if ( i != world_rank && cvalues[i] < (size_t) (world_size + 1) * 4 ) {
|
||||
fprintf(stderr, "Error in %s: count_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, cvalues[i], (size_t) (world_size + 1) * 4);
|
||||
ret = -1;
|
||||
}
|
||||
/* Check size values */
|
||||
if( i == world_rank && svalues[i] != (size_t) 0 ) {
|
||||
fprintf(stderr, "Error in %s: size_values[%d]=%zu, and should be equal to %zu.\n",
|
||||
__func__, i, svalues[i], (size_t) 0);
|
||||
ret = -1;
|
||||
} else if ( i != world_rank && svalues[i] < (size_t) (world_size * (2 * 13 * sizeof(char) + sizeof(int)) + 13 * 3 * sizeof(char) + sizeof(int)) ) {
|
||||
fprintf(stderr, "Error in %s: size_values[%d]=%zu, and should be >= %zu.\n",
|
||||
__func__, i, svalues[i], (size_t) (world_size * (2 * 13 * sizeof(char) + sizeof(int)) + 13 * 3 * sizeof(char) + sizeof(int)));
|
||||
ret = -1;
|
||||
}
|
||||
}
|
||||
/* Check One-to-all COLL values */
|
||||
pvar_o2a_read(session, &count, &size);
|
||||
if( count < (size_t) 2 ) {
|
||||
fprintf(stderr, "Error in %s: count_o2a=%zu, and should be >= %zu.\n",
|
||||
__func__, count, (size_t) 2);
|
||||
ret = -1;
|
||||
}
|
||||
if( size < (size_t) ((world_size - 1) * 13 * 2 * sizeof(char)) ) {
|
||||
fprintf(stderr, "Error in %s: size_o2a=%zu, and should be >= %zu.\n",
|
||||
__func__, size, (size_t) ((world_size - 1) * 13 * 2 * sizeof(char)));
|
||||
ret = -1;
|
||||
}
|
||||
/* Check All-to-one COLL values */
|
||||
pvar_a2o_read(session, &count, &size);
|
||||
if( count < (size_t) 2 ) {
|
||||
fprintf(stderr, "Error in %s: count_a2o=%zu, and should be >= %zu.\n",
|
||||
__func__, count, (size_t) 2);
|
||||
ret = -1;
|
||||
}
|
||||
if( size < (size_t) ((world_size - 1) * (13 * sizeof(char) + sizeof(int))) ) {
|
||||
fprintf(stderr, "Error in %s: size_a2o=%zu, and should be >= %zu.\n",
|
||||
__func__, size,
|
||||
(size_t) ((world_size - 1) * (13 * sizeof(char) + sizeof(int))));
|
||||
ret = -1;
|
||||
}
|
||||
/* Check All-to-all COLL values */
|
||||
pvar_a2a_read(session, &count, &size);
|
||||
if( count < (size_t) (world_size * 4) ) {
|
||||
fprintf(stderr, "Error in %s: count_a2a=%zu, and should be >= %zu.\n",
|
||||
__func__, count, (size_t) (world_size * 4));
|
||||
ret = -1;
|
||||
}
|
||||
if( size < (size_t) (world_size * (world_size - 1) * (2 * 13 * sizeof(char) + sizeof(int))) ) {
|
||||
fprintf(stderr, "Error in %s: size_a2a=%zu, and should be >= %zu.\n",
|
||||
__func__, size,
|
||||
(size_t) (world_size * (world_size - 1) * (2 * 13 * sizeof(char) + sizeof(int))));
|
||||
ret = -1;
|
||||
}
|
||||
if( MPI_SUCCESS == ret ) {
|
||||
fprintf(stdout, "Check COLL...[ OK ]\n");
|
||||
} else {
|
||||
fprintf(stdout, "Check COLL...[FAIL]\n");
|
||||
}
|
||||
/* Keep old PML values */
|
||||
pvar_pml_read(session, old_cvalues, old_svalues);
|
||||
/* Free arrays */
|
||||
free(cvalues);
|
||||
return ret;
|
||||
}
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
int size, i, n, to, from, world_rank;
|
||||
MPI_T_pvar_session session;
|
||||
MPI_Status status;
|
||||
char s1[20], s2[20];
|
||||
strncpy(s1, "hello world!", 13);
|
||||
|
||||
MPI_Init(NULL, NULL);
|
||||
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
|
||||
MPI_Comm_size(MPI_COMM_WORLD, &size);
|
||||
|
||||
pvar_all_init(&session, size);
|
||||
|
||||
/* first phase: exchange size times data with everyone in
|
||||
MPI_COMM_WORLD with collective operations. This phase comes
|
||||
first in order to ease the prediction of messages exchanged of
|
||||
each kind.
|
||||
*/
|
||||
char*coll_buff = malloc(2 * size * 13 * sizeof(char));
|
||||
char*coll_recv_buff = coll_buff + size * 13;
|
||||
int sum_ranks;
|
||||
for( n = 0; n < size; ++n ) {
|
||||
/* Allgather */
|
||||
memset(coll_buff, 0, size * 13 * sizeof(char));
|
||||
MPI_Allgather(s1, 13, MPI_CHAR, coll_buff, 13, MPI_CHAR, MPI_COMM_WORLD);
|
||||
for( i = 0; i < size; ++i ) {
|
||||
if( strncmp(s1, &coll_buff[i * 13], 13) ) {
|
||||
fprintf(stderr, "Error in Allgather check: received \"%s\" instead of "
|
||||
"\"hello world!\" from %d.\n", &coll_buff[i * 13], i);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
}
|
||||
/* Scatter */
|
||||
MPI_Scatter(coll_buff, 13, MPI_CHAR, s2, 13, MPI_CHAR, n, MPI_COMM_WORLD);
|
||||
if( strncmp(s1, s2, 13) ) {
|
||||
fprintf(stderr, "Error in Scatter check: received \"%s\" instead of "
|
||||
"\"hello world!\" from %d.\n", s2, n);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
/* Allreduce */
|
||||
MPI_Allreduce(&world_rank, &sum_ranks, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
|
||||
if( sum_ranks != ((size - 1) * size / 2) ) {
|
||||
fprintf(stderr, "Error in Allreduce check: sum_ranks=%d instead of %d.\n",
|
||||
sum_ranks, (size - 1) * size / 2);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
/* Alltoall */
|
||||
memset(coll_recv_buff, 0, size * 13 * sizeof(char));
|
||||
MPI_Alltoall(coll_buff, 13, MPI_CHAR, coll_recv_buff, 13, MPI_CHAR, MPI_COMM_WORLD);
|
||||
for( i = 0; i < size; ++i ) {
|
||||
if( strncmp(s1, &coll_recv_buff[i * 13], 13) ) {
|
||||
fprintf(stderr, "Error in Alltoall check: received \"%s\" instead of "
|
||||
"\"hello world!\" from %d.\n", &coll_recv_buff[i * 13], i);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
}
|
||||
/* Bcast */
|
||||
if( n == world_rank ) {
|
||||
MPI_Bcast(s1, 13, MPI_CHAR, n, MPI_COMM_WORLD);
|
||||
} else {
|
||||
MPI_Bcast(s2, 13, MPI_CHAR, n, MPI_COMM_WORLD);
|
||||
if( strncmp(s1, s2, 13) ) {
|
||||
fprintf(stderr, "Error in Bcast check: received \"%s\" instead of "
|
||||
"\"hello world!\" from %d.\n", s2, n);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
}
|
||||
/* Barrier */
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
/* Gather */
|
||||
memset(coll_buff, 0, size * 13 * sizeof(char));
|
||||
MPI_Gather(s1, 13, MPI_CHAR, coll_buff, 13, MPI_CHAR, n, MPI_COMM_WORLD);
|
||||
if( n == world_rank ) {
|
||||
for( i = 0; i < size; ++i ) {
|
||||
if( strncmp(s1, &coll_buff[i * 13], 13) ) {
|
||||
fprintf(stderr, "Error in Gather check: received \"%s\" instead of "
|
||||
"\"hello world!\" from %d.\n", &coll_buff[i * 13], i);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
}
|
||||
}
|
||||
/* Reduce */
|
||||
MPI_Reduce(&world_rank, &sum_ranks, 1, MPI_INT, MPI_SUM, n, MPI_COMM_WORLD);
|
||||
if( n == world_rank ) {
|
||||
if( sum_ranks != ((size - 1) * size / 2) ) {
|
||||
fprintf(stderr, "Error in Reduce check: sum_ranks=%d instead of %d.\n",
|
||||
sum_ranks, (size - 1) * size / 2);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
}
|
||||
}
|
||||
free(coll_buff);
|
||||
if( -1 == pvar_coll_check(session, size, world_rank) ) MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
|
||||
/* second phase: exchange size times data with everyone except self
|
||||
in MPI_COMM_WORLD with Send/Recv */
|
||||
for( n = 0; n < size; ++n ) {
|
||||
for( i = 0; i < size - 1; ++i ) {
|
||||
to = (world_rank+1+i)%size;
|
||||
from = (world_rank+size-1-i)%size;
|
||||
if(world_rank < to){
|
||||
MPI_Send(s1, 13, MPI_CHAR, to, world_rank, MPI_COMM_WORLD);
|
||||
MPI_Recv(s2, 13, MPI_CHAR, from, from, MPI_COMM_WORLD, &status);
|
||||
} else {
|
||||
MPI_Recv(s2, 13, MPI_CHAR, from, from, MPI_COMM_WORLD, &status);
|
||||
MPI_Send(s1, 13, MPI_CHAR, to, world_rank, MPI_COMM_WORLD);
|
||||
}
|
||||
if( strncmp(s2, "hello world!", 13) ) {
|
||||
fprintf(stderr, "Error in PML check: s2=\"%s\" instead of \"hello world!\".\n",
|
||||
s2);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
}
|
||||
}
|
||||
if( -1 == pvar_pml_check(session, size, world_rank) ) MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
|
||||
/* third phase: exchange size times data with everyone, including self, in
|
||||
MPI_COMM_WORLD with RMA opertations */
|
||||
char win_buff[20];
|
||||
MPI_Win win;
|
||||
MPI_Win_create(win_buff, 20, sizeof(char), MPI_INFO_NULL, MPI_COMM_WORLD, &win);
|
||||
for( n = 0; n < size; ++n ) {
|
||||
for( i = 0; i < size; ++i ) {
|
||||
MPI_Win_lock(MPI_LOCK_EXCLUSIVE, i, 0, win);
|
||||
MPI_Put(s1, 13, MPI_CHAR, i, 0, 13, MPI_CHAR, win);
|
||||
MPI_Win_unlock(i, win);
|
||||
}
|
||||
MPI_Win_lock(MPI_LOCK_EXCLUSIVE, world_rank, 0, win);
|
||||
if( strncmp(win_buff, "hello world!", 13) ) {
|
||||
fprintf(stderr, "Error in OSC check: win_buff=\"%s\" instead of \"hello world!\".\n",
|
||||
win_buff);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
MPI_Win_unlock(world_rank, win);
|
||||
for( i = 0; i < size; ++i ) {
|
||||
MPI_Win_lock(MPI_LOCK_EXCLUSIVE, i, 0, win);
|
||||
MPI_Get(s2, 13, MPI_CHAR, i, 0, 13, MPI_CHAR, win);
|
||||
MPI_Win_unlock(i, win);
|
||||
if( strncmp(s2, "hello world!", 13) ) {
|
||||
fprintf(stderr, "Error in OSC check: s2=\"%s\" instead of \"hello world!\".\n",
|
||||
s2);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
}
|
||||
}
|
||||
MPI_Win_free(&win);
|
||||
if( -1 == pvar_osc_check(session, size, world_rank) ) MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
|
||||
pvar_all_finalize(&session);
|
||||
|
||||
MPI_Finalize();
|
||||
|
||||
return EXIT_SUCCESS;
|
||||
}
|
||||
127
test/monitoring/example_reduce_count.c
Обычный файл
127
test/monitoring/example_reduce_count.c
Обычный файл
@ -0,0 +1,127 @@
|
||||
/*
|
||||
* Copyright (c) 2017 Inria. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <mpi.h>
|
||||
|
||||
static MPI_T_pvar_handle count_handle;
|
||||
static const char count_pvar_name[] = "pml_monitoring_messages_count";
|
||||
static int count_pvar_idx;
|
||||
|
||||
int main(int argc, char**argv)
|
||||
{
|
||||
int rank, size, n, to, from, tagno, MPIT_result, provided, count;
|
||||
MPI_T_pvar_session session;
|
||||
MPI_Status status;
|
||||
MPI_Request request;
|
||||
size_t*counts;
|
||||
|
||||
n = -1;
|
||||
MPI_Init(&argc, &argv);
|
||||
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
|
||||
MPI_Comm_size(MPI_COMM_WORLD, &size);
|
||||
to = (rank + 1) % size;
|
||||
from = (rank + size - 1) % size;
|
||||
tagno = 201;
|
||||
|
||||
MPIT_result = MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);
|
||||
if (MPIT_result != MPI_SUCCESS)
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
|
||||
MPIT_result = MPI_T_pvar_get_index(count_pvar_name, MPI_T_PVAR_CLASS_SIZE, &count_pvar_idx);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot find monitoring MPI_T \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_session_create(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot create a session for \"%s\" pvar\n", count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Allocating a new PVAR in a session will reset the counters */
|
||||
MPIT_result = MPI_T_pvar_handle_alloc(session, count_pvar_idx,
|
||||
MPI_COMM_WORLD, &count_handle, &count);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to allocate handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
counts = (size_t*)malloc(count * sizeof(size_t));
|
||||
|
||||
MPIT_result = MPI_T_pvar_start(session, count_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Token Ring communications */
|
||||
if (rank == 0) {
|
||||
n = 25;
|
||||
MPI_Isend(&n,1,MPI_INT,to,tagno,MPI_COMM_WORLD,&request);
|
||||
}
|
||||
while (1) {
|
||||
MPI_Irecv(&n, 1, MPI_INT, from, tagno, MPI_COMM_WORLD, &request);
|
||||
MPI_Wait(&request, &status);
|
||||
if (rank == 0) {n--;tagno++;}
|
||||
MPI_Isend(&n, 1, MPI_INT, to, tagno, MPI_COMM_WORLD, &request);
|
||||
if (rank != 0) {n--;tagno++;}
|
||||
if (n<0){
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_read(session, count_handle, counts);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to read handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/*** REDUCE ***/
|
||||
MPI_Allreduce(MPI_IN_PLACE, counts, count, MPI_UNSIGNED_LONG, MPI_MAX, MPI_COMM_WORLD);
|
||||
|
||||
if(0 == rank) {
|
||||
for(n = 0; n < count; ++n)
|
||||
printf("%zu%s", counts[n], n < count - 1 ? ", " : "\n");
|
||||
}
|
||||
|
||||
free(counts);
|
||||
|
||||
MPIT_result = MPI_T_pvar_stop(session, count_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_handle_free(session, &count_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to free handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_session_free(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot close a session for \"%s\" pvar\n", count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
(void)MPI_T_finalize();
|
||||
|
||||
MPI_Finalize();
|
||||
|
||||
return EXIT_SUCCESS;
|
||||
}
|
||||
@ -1,10 +1,12 @@
|
||||
/*
|
||||
* Copyright (c) 2013-2016 The University of Tennessee and The University
|
||||
* Copyright (c) 2013-2017 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2015 Bull SAS. All rights reserved.
|
||||
* Copyright (c) 2016 Cisco Systems, Inc. All rights reserved.
|
||||
* Copyright (c) 2016 Cisco Systems, Inc. All rights reserved.
|
||||
* Copyright (c) 2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -19,6 +21,7 @@ Designed by:
|
||||
George Bosilca <bosilca@icl.utk.edu>
|
||||
Emmanuel Jeannot <emmanuel.jeannot@inria.fr>
|
||||
Guillaume Papauré <guillaume.papaure@bull.net>
|
||||
Clément Foyer <clement.foyer@inria.fr>
|
||||
|
||||
Contact the authors for questions.
|
||||
|
||||
@ -43,8 +46,6 @@ writing 4x4 matrix to monitoring_avg.mat
|
||||
#include <stdlib.h>
|
||||
#include <mpi.h>
|
||||
#include <string.h>
|
||||
#include <stdint.h>
|
||||
#include <inttypes.h>
|
||||
|
||||
static MPI_T_pvar_session session;
|
||||
static int comm_world_size;
|
||||
@ -55,14 +56,24 @@ struct monitoring_result
|
||||
char * pvar_name;
|
||||
int pvar_idx;
|
||||
MPI_T_pvar_handle pvar_handle;
|
||||
uint64_t * vector;
|
||||
size_t * vector;
|
||||
};
|
||||
typedef struct monitoring_result monitoring_result;
|
||||
|
||||
static monitoring_result counts;
|
||||
static monitoring_result sizes;
|
||||
/* PML Sent */
|
||||
static monitoring_result pml_counts;
|
||||
static monitoring_result pml_sizes;
|
||||
/* OSC Sent */
|
||||
static monitoring_result osc_scounts;
|
||||
static monitoring_result osc_ssizes;
|
||||
/* OSC Recv */
|
||||
static monitoring_result osc_rcounts;
|
||||
static monitoring_result osc_rsizes;
|
||||
/* COLL Sent/Recv */
|
||||
static monitoring_result coll_counts;
|
||||
static monitoring_result coll_sizes;
|
||||
|
||||
static int write_mat(char *, uint64_t *, unsigned int);
|
||||
static int write_mat(char *, size_t *, unsigned int);
|
||||
static void init_monitoring_result(const char *, monitoring_result *);
|
||||
static void start_monitoring_result(monitoring_result *);
|
||||
static void stop_monitoring_result(monitoring_result *);
|
||||
@ -91,11 +102,23 @@ int MPI_Init(int* argc, char*** argv)
|
||||
PMPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
init_monitoring_result("pml_monitoring_messages_count", &counts);
|
||||
init_monitoring_result("pml_monitoring_messages_size", &sizes);
|
||||
|
||||
start_monitoring_result(&counts);
|
||||
start_monitoring_result(&sizes);
|
||||
init_monitoring_result("pml_monitoring_messages_count", &pml_counts);
|
||||
init_monitoring_result("pml_monitoring_messages_size", &pml_sizes);
|
||||
init_monitoring_result("osc_monitoring_messages_sent_count", &osc_scounts);
|
||||
init_monitoring_result("osc_monitoring_messages_sent_size", &osc_ssizes);
|
||||
init_monitoring_result("osc_monitoring_messages_recv_count", &osc_rcounts);
|
||||
init_monitoring_result("osc_monitoring_messages_recv_size", &osc_rsizes);
|
||||
init_monitoring_result("coll_monitoring_messages_count", &coll_counts);
|
||||
init_monitoring_result("coll_monitoring_messages_size", &coll_sizes);
|
||||
|
||||
start_monitoring_result(&pml_counts);
|
||||
start_monitoring_result(&pml_sizes);
|
||||
start_monitoring_result(&osc_scounts);
|
||||
start_monitoring_result(&osc_ssizes);
|
||||
start_monitoring_result(&osc_rcounts);
|
||||
start_monitoring_result(&osc_rsizes);
|
||||
start_monitoring_result(&coll_counts);
|
||||
start_monitoring_result(&coll_sizes);
|
||||
|
||||
return result;
|
||||
}
|
||||
@ -103,48 +126,143 @@ int MPI_Init(int* argc, char*** argv)
|
||||
int MPI_Finalize(void)
|
||||
{
|
||||
int result, MPIT_result;
|
||||
uint64_t * exchange_count_matrix = NULL;
|
||||
uint64_t * exchange_size_matrix = NULL;
|
||||
uint64_t * exchange_avg_size_matrix = NULL;
|
||||
size_t * exchange_count_matrix_1 = NULL;
|
||||
size_t * exchange_size_matrix_1 = NULL;
|
||||
size_t * exchange_count_matrix_2 = NULL;
|
||||
size_t * exchange_size_matrix_2 = NULL;
|
||||
size_t * exchange_all_size_matrix = NULL;
|
||||
size_t * exchange_all_count_matrix = NULL;
|
||||
size_t * exchange_all_avg_matrix = NULL;
|
||||
|
||||
stop_monitoring_result(&pml_counts);
|
||||
stop_monitoring_result(&pml_sizes);
|
||||
stop_monitoring_result(&osc_scounts);
|
||||
stop_monitoring_result(&osc_ssizes);
|
||||
stop_monitoring_result(&osc_rcounts);
|
||||
stop_monitoring_result(&osc_rsizes);
|
||||
stop_monitoring_result(&coll_counts);
|
||||
stop_monitoring_result(&coll_sizes);
|
||||
|
||||
get_monitoring_result(&pml_counts);
|
||||
get_monitoring_result(&pml_sizes);
|
||||
get_monitoring_result(&osc_scounts);
|
||||
get_monitoring_result(&osc_ssizes);
|
||||
get_monitoring_result(&osc_rcounts);
|
||||
get_monitoring_result(&osc_rsizes);
|
||||
get_monitoring_result(&coll_counts);
|
||||
get_monitoring_result(&coll_sizes);
|
||||
|
||||
if (0 == comm_world_rank) {
|
||||
exchange_count_matrix = (uint64_t *) malloc(comm_world_size * comm_world_size * sizeof(uint64_t));
|
||||
exchange_size_matrix = (uint64_t *) malloc(comm_world_size * comm_world_size * sizeof(uint64_t));
|
||||
exchange_avg_size_matrix = (uint64_t *) malloc(comm_world_size * comm_world_size * sizeof(uint64_t));
|
||||
exchange_count_matrix_1 = (size_t *) calloc(comm_world_size * comm_world_size, sizeof(size_t));
|
||||
exchange_size_matrix_1 = (size_t *) calloc(comm_world_size * comm_world_size, sizeof(size_t));
|
||||
exchange_count_matrix_2 = (size_t *) calloc(comm_world_size * comm_world_size, sizeof(size_t));
|
||||
exchange_size_matrix_2 = (size_t *) calloc(comm_world_size * comm_world_size, sizeof(size_t));
|
||||
exchange_all_size_matrix = (size_t *) calloc(comm_world_size * comm_world_size, sizeof(size_t));
|
||||
exchange_all_count_matrix = (size_t *) calloc(comm_world_size * comm_world_size, sizeof(size_t));
|
||||
exchange_all_avg_matrix = (size_t *) calloc(comm_world_size * comm_world_size, sizeof(size_t));
|
||||
}
|
||||
|
||||
stop_monitoring_result(&counts);
|
||||
stop_monitoring_result(&sizes);
|
||||
|
||||
get_monitoring_result(&counts);
|
||||
get_monitoring_result(&sizes);
|
||||
|
||||
PMPI_Gather(counts.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_count_matrix, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
PMPI_Gather(sizes.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_size_matrix, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
/* Gather PML and COLL results */
|
||||
PMPI_Gather(pml_counts.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_count_matrix_1, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
PMPI_Gather(pml_sizes.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_size_matrix_1, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
PMPI_Gather(coll_counts.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_count_matrix_2, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
PMPI_Gather(coll_sizes.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_size_matrix_2, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
|
||||
if (0 == comm_world_rank) {
|
||||
int i, j;
|
||||
|
||||
//Get the same matrix than profile2mat.pl
|
||||
for (i = 0; i < comm_world_size; ++i) {
|
||||
for (j = i + 1; j < comm_world_size; ++j) {
|
||||
exchange_count_matrix[i * comm_world_size + j] = exchange_count_matrix[j * comm_world_size + i] = (exchange_count_matrix[i * comm_world_size + j] + exchange_count_matrix[j * comm_world_size + i]) / 2;
|
||||
exchange_size_matrix[i * comm_world_size + j] = exchange_size_matrix[j * comm_world_size + i] = (exchange_size_matrix[i * comm_world_size + j] + exchange_size_matrix[j * comm_world_size + i]) / 2;
|
||||
if (exchange_count_matrix[i * comm_world_size + j] != 0)
|
||||
exchange_avg_size_matrix[i * comm_world_size + j] = exchange_avg_size_matrix[j * comm_world_size + i] = exchange_size_matrix[i * comm_world_size + j] / exchange_count_matrix[i * comm_world_size + j];
|
||||
/* Reduce PML results */
|
||||
exchange_count_matrix_1[i * comm_world_size + j] = exchange_count_matrix_1[j * comm_world_size + i] = (exchange_count_matrix_1[i * comm_world_size + j] + exchange_count_matrix_1[j * comm_world_size + i]) / 2;
|
||||
exchange_size_matrix_1[i * comm_world_size + j] = exchange_size_matrix_1[j * comm_world_size + i] = (exchange_size_matrix_1[i * comm_world_size + j] + exchange_size_matrix_1[j * comm_world_size + i]) / 2;
|
||||
if (exchange_count_matrix_1[i * comm_world_size + j] != 0)
|
||||
exchange_all_size_matrix[i * comm_world_size + j] = exchange_all_size_matrix[j * comm_world_size + i] = exchange_size_matrix_1[i * comm_world_size + j] / exchange_count_matrix_1[i * comm_world_size + j];
|
||||
|
||||
/* Reduce COLL results */
|
||||
exchange_count_matrix_2[i * comm_world_size + j] = exchange_count_matrix_2[j * comm_world_size + i] = (exchange_count_matrix_2[i * comm_world_size + j] + exchange_count_matrix_2[j * comm_world_size + i]) / 2;
|
||||
exchange_size_matrix_2[i * comm_world_size + j] = exchange_size_matrix_2[j * comm_world_size + i] = (exchange_size_matrix_2[i * comm_world_size + j] + exchange_size_matrix_2[j * comm_world_size + i]) / 2;
|
||||
if (exchange_count_matrix_2[i * comm_world_size + j] != 0)
|
||||
exchange_all_count_matrix[i * comm_world_size + j] = exchange_all_count_matrix[j * comm_world_size + i] = exchange_size_matrix_2[i * comm_world_size + j] / exchange_count_matrix_2[i * comm_world_size + j];
|
||||
}
|
||||
}
|
||||
|
||||
write_mat("monitoring_msg.mat", exchange_count_matrix, comm_world_size);
|
||||
write_mat("monitoring_size.mat", exchange_size_matrix, comm_world_size);
|
||||
write_mat("monitoring_avg.mat", exchange_avg_size_matrix, comm_world_size);
|
||||
/* Write PML matrices */
|
||||
write_mat("monitoring_pml_msg.mat", exchange_count_matrix_1, comm_world_size);
|
||||
write_mat("monitoring_pml_size.mat", exchange_size_matrix_1, comm_world_size);
|
||||
write_mat("monitoring_pml_avg.mat", exchange_all_size_matrix, comm_world_size);
|
||||
|
||||
/* Write COLL matrices */
|
||||
write_mat("monitoring_coll_msg.mat", exchange_count_matrix_2, comm_world_size);
|
||||
write_mat("monitoring_coll_size.mat", exchange_size_matrix_2, comm_world_size);
|
||||
write_mat("monitoring_coll_avg.mat", exchange_all_count_matrix, comm_world_size);
|
||||
|
||||
/* Aggregate PML and COLL in ALL matrices */
|
||||
for (i = 0; i < comm_world_size; ++i) {
|
||||
for (j = i + 1; j < comm_world_size; ++j) {
|
||||
exchange_all_size_matrix[i * comm_world_size + j] = exchange_all_size_matrix[j * comm_world_size + i] = exchange_size_matrix_1[i * comm_world_size + j] + exchange_size_matrix_2[i * comm_world_size + j];
|
||||
exchange_all_count_matrix[i * comm_world_size + j] = exchange_all_count_matrix[j * comm_world_size + i] = exchange_count_matrix_1[i * comm_world_size + j] + exchange_count_matrix_2[i * comm_world_size + j];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
free(exchange_count_matrix);
|
||||
free(exchange_size_matrix);
|
||||
free(exchange_avg_size_matrix);
|
||||
destroy_monitoring_result(&counts);
|
||||
destroy_monitoring_result(&sizes);
|
||||
/* Gather OSC results */
|
||||
PMPI_Gather(osc_scounts.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_count_matrix_1, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
PMPI_Gather(osc_ssizes.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_size_matrix_1, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
PMPI_Gather(osc_rcounts.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_count_matrix_2, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
PMPI_Gather(osc_rsizes.vector, comm_world_size, MPI_UNSIGNED_LONG, exchange_size_matrix_2, comm_world_size, MPI_UNSIGNED_LONG, 0, MPI_COMM_WORLD);
|
||||
|
||||
if (0 == comm_world_rank) {
|
||||
int i, j;
|
||||
|
||||
for (i = 0; i < comm_world_size; ++i) {
|
||||
for (j = i + 1; j < comm_world_size; ++j) {
|
||||
/* Reduce OSC results */
|
||||
exchange_count_matrix_1[i * comm_world_size + j] = exchange_count_matrix_1[j * comm_world_size + i] = (exchange_count_matrix_1[i * comm_world_size + j] + exchange_count_matrix_1[j * comm_world_size + i] + exchange_count_matrix_2[i * comm_world_size + j] + exchange_count_matrix_2[j * comm_world_size + i]) / 2;
|
||||
exchange_size_matrix_1[i * comm_world_size + j] = exchange_size_matrix_1[j * comm_world_size + i] = (exchange_size_matrix_1[i * comm_world_size + j] + exchange_size_matrix_1[j * comm_world_size + i] + exchange_size_matrix_2[i * comm_world_size + j] + exchange_size_matrix_2[j * comm_world_size + i]) / 2;
|
||||
if (exchange_count_matrix_1[i * comm_world_size + j] != 0)
|
||||
exchange_all_avg_matrix[i * comm_world_size + j] = exchange_all_avg_matrix[j * comm_world_size + i] = exchange_size_matrix_1[i * comm_world_size + j] / exchange_count_matrix_1[i * comm_world_size + j];
|
||||
}
|
||||
}
|
||||
|
||||
/* Write OSC matrices */
|
||||
write_mat("monitoring_osc_msg.mat", exchange_count_matrix_1, comm_world_size);
|
||||
write_mat("monitoring_osc_size.mat", exchange_size_matrix_1, comm_world_size);
|
||||
write_mat("monitoring_osc_avg.mat", exchange_all_avg_matrix, comm_world_size);
|
||||
|
||||
/* Aggregate OSC in ALL matrices and compute AVG */
|
||||
for (i = 0; i < comm_world_size; ++i) {
|
||||
for (j = i + 1; j < comm_world_size; ++j) {
|
||||
exchange_all_size_matrix[i * comm_world_size + j] = exchange_all_size_matrix[j * comm_world_size + i] += exchange_size_matrix_1[i * comm_world_size + j];
|
||||
exchange_all_count_matrix[i * comm_world_size + j] = exchange_all_count_matrix[j * comm_world_size + i] += exchange_count_matrix_1[i * comm_world_size + j];
|
||||
if (exchange_all_count_matrix[i * comm_world_size + j] != 0)
|
||||
exchange_all_avg_matrix[i * comm_world_size + j] = exchange_all_avg_matrix[j * comm_world_size + i] = exchange_all_size_matrix[i * comm_world_size + j] / exchange_all_count_matrix[i * comm_world_size + j];
|
||||
}
|
||||
}
|
||||
|
||||
/* Write ALL matrices */
|
||||
write_mat("monitoring_all_msg.mat", exchange_all_count_matrix, comm_world_size);
|
||||
write_mat("monitoring_all_size.mat", exchange_all_size_matrix, comm_world_size);
|
||||
write_mat("monitoring_all_avg.mat", exchange_all_avg_matrix, comm_world_size);
|
||||
|
||||
/* Free matrices */
|
||||
free(exchange_count_matrix_1);
|
||||
free(exchange_size_matrix_1);
|
||||
free(exchange_count_matrix_2);
|
||||
free(exchange_size_matrix_2);
|
||||
free(exchange_all_count_matrix);
|
||||
free(exchange_all_size_matrix);
|
||||
free(exchange_all_avg_matrix);
|
||||
}
|
||||
|
||||
destroy_monitoring_result(&pml_counts);
|
||||
destroy_monitoring_result(&pml_sizes);
|
||||
destroy_monitoring_result(&osc_scounts);
|
||||
destroy_monitoring_result(&osc_ssizes);
|
||||
destroy_monitoring_result(&osc_rcounts);
|
||||
destroy_monitoring_result(&osc_rsizes);
|
||||
destroy_monitoring_result(&coll_counts);
|
||||
destroy_monitoring_result(&coll_sizes);
|
||||
|
||||
MPIT_result = MPI_T_pvar_session_free(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
@ -186,7 +304,7 @@ void init_monitoring_result(const char * pvar_name, monitoring_result * res)
|
||||
PMPI_Abort(MPI_COMM_WORLD, count);
|
||||
}
|
||||
|
||||
res->vector = (uint64_t *) malloc(comm_world_size * sizeof(uint64_t));
|
||||
res->vector = (size_t *) malloc(comm_world_size * sizeof(size_t));
|
||||
}
|
||||
|
||||
void start_monitoring_result(monitoring_result * res)
|
||||
@ -236,7 +354,7 @@ void destroy_monitoring_result(monitoring_result * res)
|
||||
free(res->vector);
|
||||
}
|
||||
|
||||
int write_mat(char * filename, uint64_t * mat, unsigned int dim)
|
||||
int write_mat(char * filename, size_t * mat, unsigned int dim)
|
||||
{
|
||||
FILE *matrix_file;
|
||||
int i, j;
|
||||
@ -251,7 +369,7 @@ int write_mat(char * filename, uint64_t * mat, unsigned int dim)
|
||||
|
||||
for (i = 0; i < comm_world_size; ++i) {
|
||||
for (j = 0; j < comm_world_size; ++j) {
|
||||
fprintf(matrix_file, "%" PRIu64 " ", mat[i * comm_world_size + j]);
|
||||
fprintf(matrix_file, "%zu ", mat[i * comm_world_size + j]);
|
||||
}
|
||||
fprintf(matrix_file, "\n");
|
||||
}
|
||||
@ -260,3 +378,67 @@ int write_mat(char * filename, uint64_t * mat, unsigned int dim)
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* MPI binding for fortran
|
||||
*/
|
||||
|
||||
#include <stdbool.h>
|
||||
#include "ompi_config.h"
|
||||
#include "opal/threads/thread_usage.h"
|
||||
#include "ompi/mpi/fortran/base/constants.h"
|
||||
#include "ompi/mpi/fortran/base/fint_2_int.h"
|
||||
|
||||
void monitoring_prof_mpi_init_f2c( MPI_Fint * );
|
||||
void monitoring_prof_mpi_finalize_f2c( MPI_Fint * );
|
||||
|
||||
void monitoring_prof_mpi_init_f2c( MPI_Fint *ierr ) {
|
||||
int c_ierr;
|
||||
int argc = 0;
|
||||
char ** argv = NULL;
|
||||
|
||||
c_ierr = MPI_Init(&argc, &argv);
|
||||
if (NULL != ierr) *ierr = OMPI_INT_2_FINT(c_ierr);
|
||||
}
|
||||
|
||||
void monitoring_prof_mpi_finalize_f2c( MPI_Fint *ierr ) {
|
||||
int c_ierr;
|
||||
|
||||
c_ierr = MPI_Finalize();
|
||||
if (NULL != ierr) *ierr = OMPI_INT_2_FINT(c_ierr);
|
||||
}
|
||||
|
||||
#if OPAL_HAVE_WEAK_SYMBOLS
|
||||
#pragma weak MPI_INIT = monitoring_prof_mpi_init_f2c
|
||||
#pragma weak mpi_init = monitoring_prof_mpi_init_f2c
|
||||
#pragma weak mpi_init_ = monitoring_prof_mpi_init_f2c
|
||||
#pragma weak mpi_init__ = monitoring_prof_mpi_init_f2c
|
||||
#pragma weak MPI_Init_f = monitoring_prof_mpi_init_f2c
|
||||
#pragma weak MPI_Init_f08 = monitoring_prof_mpi_init_f2c
|
||||
|
||||
#pragma weak MPI_FINALIZE = monitoring_prof_mpi_finalize_f2c
|
||||
#pragma weak mpi_finalize = monitoring_prof_mpi_finalize_f2c
|
||||
#pragma weak mpi_finalize_ = monitoring_prof_mpi_finalize_f2c
|
||||
#pragma weak mpi_finalize__ = monitoring_prof_mpi_finalize_f2c
|
||||
#pragma weak MPI_Finalize_f = monitoring_prof_mpi_finalize_f2c
|
||||
#pragma weak MPI_Finalize_f08 = monitoring_prof_mpi_finalize_f2c
|
||||
#elif OMPI_BUILD_FORTRAN_BINDINGS
|
||||
#define OMPI_F77_PROTOTYPES_MPI_H
|
||||
#include "ompi/mpi/fortran/mpif-h/bindings.h"
|
||||
|
||||
OMPI_GENERATE_F77_BINDINGS (MPI_INIT,
|
||||
mpi_init,
|
||||
mpi_init_,
|
||||
mpi_init__,
|
||||
monitoring_prof_mpi_init_f2c,
|
||||
(MPI_Fint *ierr),
|
||||
(ierr) )
|
||||
|
||||
OMPI_GENERATE_F77_BINDINGS (MPI_FINALIZE,
|
||||
mpi_finalize,
|
||||
mpi_finalize_,
|
||||
mpi_finalize__,
|
||||
monitoring_prof_mpi_finalize_f2c,
|
||||
(MPI_Fint *ierr),
|
||||
(ierr) )
|
||||
#endif
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
* Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
* Copyright (c) 2013-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2015 Cisco Systems, Inc. All rights reserved.
|
||||
* Copyright (c) 2016 Intel, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
@ -15,243 +15,362 @@
|
||||
/*
|
||||
pml monitoring tester.
|
||||
|
||||
Designed by George Bosilca <bosilca@icl.utk.edu> and Emmanuel Jeannot <emmanuel.jeannot@inria.fr>
|
||||
Designed by George Bosilca <bosilca@icl.utk.edu> Emmanuel Jeannot <emmanuel.jeannot@inria.fr> and Clément Foyer <clement.foyer@inria.fr>
|
||||
Contact the authors for questions.
|
||||
|
||||
To be run as:
|
||||
To options are available for this test, with/without MPI_Tools, and with/without RMA operations. The default mode is without MPI_Tools, and with RMA operations.
|
||||
To enable the MPI_Tools use, add "--with-mpit" as an application parameter.
|
||||
To disable the RMA operations testing, add "--without-rma" as an application parameter.
|
||||
|
||||
mpirun -np 4 --mca pml_monitoring_enable 2 ./monitoring_test
|
||||
pm
|
||||
Then, the output should be:
|
||||
To be run as (without using MPI_Tool):
|
||||
|
||||
flushing to ./prof/phase_1_2.prof
|
||||
flushing to ./prof/phase_1_0.prof
|
||||
flushing to ./prof/phase_1_3.prof
|
||||
flushing to ./prof/phase_2_1.prof
|
||||
flushing to ./prof/phase_2_3.prof
|
||||
flushing to ./prof/phase_2_0.prof
|
||||
flushing to ./prof/phase_2_2.prof
|
||||
I 0 1 108 bytes 27 msgs sent
|
||||
E 0 1 1012 bytes 30 msgs sent
|
||||
E 0 2 23052 bytes 61 msgs sent
|
||||
I 1 2 104 bytes 26 msgs sent
|
||||
I 1 3 208 bytes 52 msgs sent
|
||||
E 1 0 860 bytes 24 msgs sent
|
||||
E 1 3 2552 bytes 56 msgs sent
|
||||
I 2 3 104 bytes 26 msgs sent
|
||||
E 2 0 22804 bytes 49 msgs sent
|
||||
E 2 3 860 bytes 24 msgs sent
|
||||
I 3 0 104 bytes 26 msgs sent
|
||||
I 3 1 204 bytes 51 msgs sent
|
||||
E 3 1 2304 bytes 44 msgs sent
|
||||
E 3 2 860 bytes 24 msgs sent
|
||||
mpirun -np 4 --mca pml_monitoring_enable 2 --mca pml_monitoring_enable_output 3 --mca pml_monitoring_filename prof/output ./monitoring_test
|
||||
|
||||
or as
|
||||
with the results being, as an example:
|
||||
output.1.prof
|
||||
# POINT TO POINT
|
||||
E 1 2 104 bytes 26 msgs sent 0,0,0,26,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
|
||||
E 1 3 208 bytes 52 msgs sent 8,0,0,65,1,5,2,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
|
||||
I 1 0 140 bytes 27 msgs sent
|
||||
I 1 2 2068 bytes 1 msgs sent
|
||||
I 1 3 2256 bytes 31 msgs sent
|
||||
# OSC
|
||||
S 1 0 0 bytes 1 msgs sent
|
||||
R 1 0 40960 bytes 1 msgs sent
|
||||
S 1 2 40960 bytes 1 msgs sent
|
||||
# COLLECTIVES
|
||||
C 1 0 140 bytes 27 msgs sent
|
||||
C 1 2 140 bytes 27 msgs sent
|
||||
C 1 3 140 bytes 27 msgs sent
|
||||
D MPI COMMUNICATOR 4 DUP FROM 0 procs: 0,1,2,3
|
||||
O2A 1 0 bytes 0 msgs sent
|
||||
A2O 1 0 bytes 0 msgs sent
|
||||
A2A 1 276 bytes 15 msgs sent
|
||||
D MPI_COMM_WORLD procs: 0,1,2,3
|
||||
O2A 1 0 bytes 0 msgs sent
|
||||
A2O 1 0 bytes 0 msgs sent
|
||||
A2A 1 96 bytes 9 msgs sent
|
||||
D MPI COMMUNICATOR 5 SPLIT_TYPE FROM 4 procs: 0,1,2,3
|
||||
O2A 1 0 bytes 0 msgs sent
|
||||
A2O 1 0 bytes 0 msgs sent
|
||||
A2A 1 48 bytes 3 msgs sent
|
||||
D MPI COMMUNICATOR 3 SPLIT FROM 0 procs: 1,3
|
||||
O2A 1 0 bytes 0 msgs sent
|
||||
A2O 1 0 bytes 0 msgs sent
|
||||
A2A 1 0 bytes 0 msgs sent
|
||||
|
||||
mpirun -np 4 --mca pml_monitoring_enable 1 ./monitoring_test
|
||||
|
||||
for an output as:
|
||||
|
||||
flushing to ./prof/phase_1_1.prof
|
||||
flushing to ./prof/phase_1_0.prof
|
||||
flushing to ./prof/phase_1_2.prof
|
||||
flushing to ./prof/phase_1_3.prof
|
||||
flushing to ./prof/phase_2_1.prof
|
||||
flushing to ./prof/phase_2_3.prof
|
||||
flushing to ./prof/phase_2_2.prof
|
||||
flushing to ./prof/phase_2_0.prof
|
||||
I 0 1 1120 bytes 57 msgs sent
|
||||
I 0 2 23052 bytes 61 msgs sent
|
||||
I 1 0 860 bytes 24 msgs sent
|
||||
I 1 2 104 bytes 26 msgs sent
|
||||
I 1 3 2760 bytes 108 msgs sent
|
||||
I 2 0 22804 bytes 49 msgs sent
|
||||
I 2 3 964 bytes 50 msgs sent
|
||||
I 3 0 104 bytes 26 msgs sent
|
||||
I 3 1 2508 bytes 95 msgs sent
|
||||
I 3 2 860 bytes 24 msgs sent
|
||||
*/
|
||||
|
||||
|
||||
|
||||
#include <stdio.h>
|
||||
#include "mpi.h"
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
|
||||
static MPI_T_pvar_handle flush_handle;
|
||||
static const char flush_pvar_name[] = "pml_monitoring_flush";
|
||||
static const void*nullbuf = NULL;
|
||||
static int flush_pvar_idx;
|
||||
static int with_mpit = 0;
|
||||
static int with_rma = 1;
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
int rank, size, n, to, from, tagno, MPIT_result, provided, count;
|
||||
int rank, size, n, to, from, tagno, MPIT_result, provided, count, world_rank;
|
||||
MPI_T_pvar_session session;
|
||||
MPI_Status status;
|
||||
MPI_Comm newcomm;
|
||||
MPI_Request request;
|
||||
char filename[1024];
|
||||
|
||||
|
||||
for ( int arg_it = 1; argc > 1 && arg_it < argc; ++arg_it ) {
|
||||
if( 0 == strcmp(argv[arg_it], "--with-mpit") ) {
|
||||
with_mpit = 1;
|
||||
printf("enable MPIT support\n");
|
||||
} else if( 0 == strcmp(argv[arg_it], "--without-rma") ) {
|
||||
with_rma = 0;
|
||||
printf("disable RMA testing\n");
|
||||
}
|
||||
}
|
||||
|
||||
/* first phase : make a token circulated in MPI_COMM_WORLD */
|
||||
n = -1;
|
||||
MPI_Init(&argc, &argv);
|
||||
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
|
||||
MPI_Init(NULL, NULL);
|
||||
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
|
||||
MPI_Comm_size(MPI_COMM_WORLD, &size);
|
||||
rank = world_rank;
|
||||
to = (rank + 1) % size;
|
||||
from = (rank - 1) % size;
|
||||
tagno = 201;
|
||||
|
||||
MPIT_result = MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);
|
||||
if (MPIT_result != MPI_SUCCESS)
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
if( with_mpit ) {
|
||||
MPIT_result = MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);
|
||||
if (MPIT_result != MPI_SUCCESS)
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
|
||||
MPIT_result = MPI_T_pvar_get_index(flush_pvar_name, MPI_T_PVAR_CLASS_GENERIC, &flush_pvar_idx);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot find monitoring MPI_T \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_get_index(flush_pvar_name, MPI_T_PVAR_CLASS_GENERIC, &flush_pvar_idx);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot find monitoring MPI_T \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_session_create(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot create a session for \"%s\" pvar\n", flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_session_create(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot create a session for \"%s\" pvar\n", flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Allocating a new PVAR in a session will reset the counters */
|
||||
MPIT_result = MPI_T_pvar_handle_alloc(session, flush_pvar_idx,
|
||||
MPI_COMM_WORLD, &flush_handle, &count);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to allocate handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
/* Allocating a new PVAR in a session will reset the counters */
|
||||
MPIT_result = MPI_T_pvar_handle_alloc(session, flush_pvar_idx,
|
||||
MPI_COMM_WORLD, &flush_handle, &count);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to allocate handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_start(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
MPIT_result = MPI_T_pvar_start(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
}
|
||||
|
||||
if (rank == 0) {
|
||||
n = 25;
|
||||
MPI_Isend(&n,1,MPI_INT,to,tagno,MPI_COMM_WORLD,&request);
|
||||
MPI_Send(&n,1,MPI_INT,to,tagno,MPI_COMM_WORLD);
|
||||
}
|
||||
while (1) {
|
||||
MPI_Irecv(&n,1,MPI_INT,from,tagno,MPI_COMM_WORLD, &request);
|
||||
MPI_Wait(&request,&status);
|
||||
MPI_Recv(&n, 1, MPI_INT, from, tagno, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
|
||||
if (rank == 0) {n--;tagno++;}
|
||||
MPI_Isend(&n,1,MPI_INT,to,tagno,MPI_COMM_WORLD, &request);
|
||||
MPI_Send(&n, 1, MPI_INT, to, tagno, MPI_COMM_WORLD);
|
||||
if (rank != 0) {n--;tagno++;}
|
||||
if (n<0){
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/* Build one file per processes
|
||||
Every thing that has been monitored by each
|
||||
process since the last flush will be output in filename */
|
||||
if( with_mpit ) {
|
||||
/* Build one file per processes
|
||||
Every thing that has been monitored by each
|
||||
process since the last flush will be output in filename */
|
||||
/*
|
||||
Requires directory prof to be created.
|
||||
Filename format should display the phase number
|
||||
and the process rank for ease of parsing with
|
||||
aggregate_profile.pl script
|
||||
*/
|
||||
sprintf(filename, "prof/phase_1");
|
||||
|
||||
/*
|
||||
Requires directory prof to be created.
|
||||
Filename format should display the phase number
|
||||
and the process rank for ease of parsing with
|
||||
aggregate_profile.pl script
|
||||
*/
|
||||
sprintf(filename,"prof/phase_1_%d.prof",rank);
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, filename) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s\n", rank, filename);
|
||||
}
|
||||
/* Force the writing of the monitoring data */
|
||||
MPIT_result = MPI_T_pvar_stop(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, filename) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s.%d.prof\n",
|
||||
world_rank, filename, world_rank);
|
||||
}
|
||||
/* Force the writing of the monitoring data */
|
||||
MPIT_result = MPI_T_pvar_stop(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_start(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
/* Don't set a filename. If we stop the session before setting it, then no output ile
|
||||
* will be generated.
|
||||
*/
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, NULL) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s\n", rank, filename);
|
||||
MPIT_result = MPI_T_pvar_start(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
/* Don't set a filename. If we stop the session before setting it, then no output file
|
||||
* will be generated.
|
||||
*/
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, (void*)&nullbuf) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s\n", world_rank, filename);
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
Second phase. Work with different communicators.
|
||||
even ranls will circulate a token
|
||||
while odd ranks wil perform a all_to_all
|
||||
even ranks will circulate a token
|
||||
while odd ranks will perform a all_to_all
|
||||
*/
|
||||
MPI_Comm_split(MPI_COMM_WORLD, rank%2, rank, &newcomm);
|
||||
|
||||
/* the filename for flushing monitoring now uses 2 as phase number! */
|
||||
sprintf(filename, "prof/phase_2_%d.prof", rank);
|
||||
|
||||
if(rank%2){ /*even ranks (in COMM_WORD) circulate a token*/
|
||||
if(rank%2){ /*odd ranks (in COMM_WORD) circulate a token*/
|
||||
MPI_Comm_rank(newcomm, &rank);
|
||||
MPI_Comm_size(newcomm, &size);
|
||||
if( size > 1 ) {
|
||||
to = (rank + 1) % size;;
|
||||
from = (rank - 1) % size ;
|
||||
to = (rank + 1) % size;
|
||||
from = (rank - 1) % size;
|
||||
tagno = 201;
|
||||
if (rank == 0){
|
||||
n = 50;
|
||||
MPI_Send(&n, 1, MPI_INT, to, tagno, newcomm);
|
||||
}
|
||||
while (1){
|
||||
MPI_Recv(&n, 1, MPI_INT, from, tagno, newcomm, &status);
|
||||
MPI_Recv(&n, 1, MPI_INT, from, tagno, newcomm, MPI_STATUS_IGNORE);
|
||||
if (rank == 0) {n--; tagno++;}
|
||||
MPI_Send(&n, 1, MPI_INT, to, tagno, newcomm);
|
||||
if (rank != 0) {n--; tagno++;}
|
||||
if (n<0){
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, filename) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s\n", rank, filename);
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
} else { /*odd ranks (in COMM_WORD) will perform a all_to_all and a barrier*/
|
||||
} else { /*even ranks (in COMM_WORD) will perform a all_to_all and a barrier*/
|
||||
int send_buff[10240];
|
||||
int recv_buff[10240];
|
||||
MPI_Comm newcomm2;
|
||||
MPI_Comm_rank(newcomm, &rank);
|
||||
MPI_Comm_size(newcomm, &size);
|
||||
MPI_Alltoall(send_buff, 10240/size, MPI_INT, recv_buff, 10240/size, MPI_INT, newcomm);
|
||||
MPI_Comm_split(newcomm, rank%2, rank, &newcomm);
|
||||
MPI_Barrier(newcomm);
|
||||
MPI_Comm_split(newcomm, rank%2, rank, &newcomm2);
|
||||
MPI_Barrier(newcomm2);
|
||||
MPI_Comm_free(&newcomm2);
|
||||
}
|
||||
|
||||
if( with_mpit ) {
|
||||
/* Build one file per processes
|
||||
Every thing that has been monitored by each
|
||||
process since the last flush will be output in filename */
|
||||
/*
|
||||
Requires directory prof to be created.
|
||||
Filename format should display the phase number
|
||||
and the process rank for ease of parsing with
|
||||
aggregate_profile.pl script
|
||||
*/
|
||||
sprintf(filename, "prof/phase_2");
|
||||
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, filename) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s\n", rank, filename);
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s.%d.prof\n",
|
||||
world_rank, filename, world_rank);
|
||||
}
|
||||
|
||||
/* Force the writing of the monitoring data */
|
||||
MPIT_result = MPI_T_pvar_stop(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_start(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
/* Don't set a filename. If we stop the session before setting it, then no output
|
||||
* will be generated.
|
||||
*/
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, (void*)&nullbuf ) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s\n", world_rank, filename);
|
||||
}
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_stop(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
if( with_rma ) {
|
||||
MPI_Win win;
|
||||
int rs_buff[10240];
|
||||
int win_buff[10240];
|
||||
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
|
||||
MPI_Comm_size(MPI_COMM_WORLD, &size);
|
||||
to = (rank + 1) % size;
|
||||
from = (rank + size - 1) % size;
|
||||
for( int v = 0; v < 10240; ++v )
|
||||
rs_buff[v] = win_buff[v] = rank;
|
||||
|
||||
MPI_Win_create(win_buff, 10240 * sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &win);
|
||||
MPI_Win_fence(MPI_MODE_NOPRECEDE, win);
|
||||
if( rank%2 ) {
|
||||
MPI_Win_fence(MPI_MODE_NOSTORE | MPI_MODE_NOPUT, win);
|
||||
MPI_Get(rs_buff, 10240, MPI_INT, from, 0, 10240, MPI_INT, win);
|
||||
} else {
|
||||
MPI_Put(rs_buff, 10240, MPI_INT, to, 0, 10240, MPI_INT, win);
|
||||
MPI_Win_fence(MPI_MODE_NOSTORE | MPI_MODE_NOPUT, win);
|
||||
}
|
||||
MPI_Win_fence(MPI_MODE_NOSUCCEED, win);
|
||||
|
||||
for( int v = 0; v < 10240; ++v )
|
||||
if( rs_buff[v] != win_buff[v] && ((rank%2 && rs_buff[v] != from) || (!(rank%2) && rs_buff[v] != rank)) ) {
|
||||
printf("Error on checking exchanged values: %s_buff[%d] == %d instead of %d\n",
|
||||
rank%2 ? "rs" : "win", v, rs_buff[v], rank%2 ? from : rank);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
|
||||
MPI_Group world_group, newcomm_group, distant_group;
|
||||
MPI_Comm_group(MPI_COMM_WORLD, &world_group);
|
||||
MPI_Comm_group(newcomm, &newcomm_group);
|
||||
MPI_Group_difference(world_group, newcomm_group, &distant_group);
|
||||
if( rank%2 ) {
|
||||
MPI_Win_post(distant_group, 0, win);
|
||||
MPI_Win_wait(win);
|
||||
/* Check recieved values */
|
||||
for( int v = 0; v < 10240; ++v )
|
||||
if( from != win_buff[v] ) {
|
||||
printf("Error on checking exchanged values: win_buff[%d] == %d instead of %d\n",
|
||||
v, win_buff[v], from);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
} else {
|
||||
MPI_Win_start(distant_group, 0, win);
|
||||
MPI_Put(rs_buff, 10240, MPI_INT, to, 0, 10240, MPI_INT, win);
|
||||
MPI_Win_complete(win);
|
||||
}
|
||||
MPI_Group_free(&world_group);
|
||||
MPI_Group_free(&newcomm_group);
|
||||
MPI_Group_free(&distant_group);
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
|
||||
for( int v = 0; v < 10240; ++v ) rs_buff[v] = rank;
|
||||
|
||||
MPI_Win_lock(MPI_LOCK_EXCLUSIVE, to, 0, win);
|
||||
MPI_Put(rs_buff, 10240, MPI_INT, to, 0, 10240, MPI_INT, win);
|
||||
MPI_Win_unlock(to, win);
|
||||
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
|
||||
/* Check recieved values */
|
||||
for( int v = 0; v < 10240; ++v )
|
||||
if( from != win_buff[v] ) {
|
||||
printf("Error on checking exchanged values: win_buff[%d] == %d instead of %d\n",
|
||||
v, win_buff[v], from);
|
||||
MPI_Abort(MPI_COMM_WORLD, -1);
|
||||
}
|
||||
|
||||
MPI_Win_free(&win);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_handle_free(session, &flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to free handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
if( with_mpit ) {
|
||||
/* the filename for flushing monitoring now uses 3 as phase number! */
|
||||
sprintf(filename, "prof/phase_3");
|
||||
|
||||
if( MPI_SUCCESS != MPI_T_pvar_write(session, flush_handle, filename) ) {
|
||||
fprintf(stderr, "Process %d cannot save monitoring in %s.%d.prof\n",
|
||||
world_rank, filename, world_rank);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_stop(session, flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_handle_free(session, &flush_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to free handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_session_free(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot close a session for \"%s\" pvar\n", flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
(void)MPI_T_finalize();
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_session_free(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot close a session for \"%s\" pvar\n", flush_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
(void)PMPI_T_finalize();
|
||||
|
||||
MPI_Comm_free(&newcomm);
|
||||
/* Now, in MPI_Finalize(), the pml_monitoring library outputs, in
|
||||
STDERR, the aggregated recorded monitoring of all the phases*/
|
||||
MPI_Finalize();
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
# Copyright (c) 2013-2015 The University of Tennessee and The University
|
||||
# of Tennessee Research Foundation. All rights
|
||||
# reserved.
|
||||
# Copyright (c) 2013-2015 Inria. All rights reserved.
|
||||
# Copyright (c) 2013-2016 Inria. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
@ -35,9 +35,11 @@ if($#ARGV < 0){
|
||||
$filename=$ARGV[0];
|
||||
}
|
||||
|
||||
profile($filename,"I|E","all");
|
||||
profile($filename,"I|E|S|R|C","all");
|
||||
if ( profile($filename,"E","external") ){
|
||||
profile($filename,"I","internal");
|
||||
profile($filename,"I","internal");
|
||||
profile($filename,"S|R","osc");
|
||||
profile($filename,"C","coll");
|
||||
}
|
||||
|
||||
sub profile{
|
||||
|
||||
294
test/monitoring/test_overhead.c
Обычный файл
294
test/monitoring/test_overhead.c
Обычный файл
@ -0,0 +1,294 @@
|
||||
/*
|
||||
* Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
* Copyright (c) 2017 Research Organization for Information Science
|
||||
* and Technology (RIST). All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
/*
|
||||
Measurement for thze pml_monitoring component overhead
|
||||
|
||||
Designed by Clement Foyer <clement.foyer@inria.fr>
|
||||
Contact the authors for questions.
|
||||
|
||||
To be run as:
|
||||
|
||||
|
||||
*/
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <sys/time.h>
|
||||
#include <time.h>
|
||||
#include <string.h>
|
||||
#include "mpi.h"
|
||||
|
||||
#define NB_ITER 1000
|
||||
#define FULL_NB_ITER (size_world * NB_ITER)
|
||||
#define MAX_SIZE (1024 * 1024 * 1.4)
|
||||
#define NB_OPS 6
|
||||
|
||||
static int rank_world = -1;
|
||||
static int size_world = 0;
|
||||
static int to = -1;
|
||||
static int from = -1;
|
||||
static MPI_Win win = MPI_WIN_NULL;
|
||||
|
||||
/* Sorting results */
|
||||
static int comp_double(const void*_a, const void*_b)
|
||||
{
|
||||
const double*a = _a;
|
||||
const double*b = _b;
|
||||
if(*a < *b)
|
||||
return -1;
|
||||
else if(*a > *b)
|
||||
return 1;
|
||||
else
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Timing */
|
||||
static inline void get_tick(struct timespec*t)
|
||||
{
|
||||
#if defined(__bg__)
|
||||
# define CLOCK_TYPE CLOCK_REALTIME
|
||||
#elif defined(CLOCK_MONOTONIC_RAW)
|
||||
# define CLOCK_TYPE CLOCK_MONOTONIC_RAW
|
||||
#elif defined(CLOCK_MONOTONIC)
|
||||
# define CLOCK_TYPE CLOCK_MONOTONIC
|
||||
#endif
|
||||
#if defined(CLOCK_TYPE)
|
||||
clock_gettime(CLOCK_TYPE, t);
|
||||
#else
|
||||
struct timeval tv;
|
||||
gettimeofday(&tv, NULL);
|
||||
t->tv_sec = tv.tv_sec;
|
||||
t->tv_nsec = tv.tv_usec * 1000;
|
||||
#endif
|
||||
}
|
||||
static inline double timing_delay(const struct timespec*const t1, const struct timespec*const t2)
|
||||
{
|
||||
const double delay = 1000000.0 * (t2->tv_sec - t1->tv_sec) + (t2->tv_nsec - t1->tv_nsec) / 1000.0;
|
||||
return delay;
|
||||
}
|
||||
|
||||
/* Operations */
|
||||
static inline void op_send(double*res, void*sbuf, int size, int tagno, void*rbuf) {
|
||||
MPI_Request request;
|
||||
struct timespec start, end;
|
||||
|
||||
/* Post to be sure no unexpected message will be generated */
|
||||
MPI_Irecv(rbuf, size, MPI_BYTE, from, tagno, MPI_COMM_WORLD, &request);
|
||||
|
||||
/* Token ring to synchronize */
|
||||
/* We send to the sender to make him know we are ready to
|
||||
receive (even for non-eager mode sending) */
|
||||
if( 0 == rank_world ) {
|
||||
MPI_Send(NULL, 0, MPI_BYTE, from, 100, MPI_COMM_WORLD);
|
||||
MPI_Recv(NULL, 0, MPI_BYTE, to, 100, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
|
||||
} else {
|
||||
MPI_Recv(NULL, 0, MPI_BYTE, to, 100, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
|
||||
MPI_Send(NULL, 0, MPI_BYTE, from, 100, MPI_COMM_WORLD);
|
||||
}
|
||||
|
||||
/* do monitored operation */
|
||||
get_tick(&start);
|
||||
MPI_Send(sbuf, size, MPI_BYTE, to, tagno, MPI_COMM_WORLD);
|
||||
get_tick(&end);
|
||||
|
||||
MPI_Wait(&request, MPI_STATUS_IGNORE);
|
||||
*res = timing_delay(&start, &end);
|
||||
}
|
||||
|
||||
static inline void op_send_pingpong(double*res, void*sbuf, int size, int tagno, void*rbuf) {
|
||||
struct timespec start, end;
|
||||
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
|
||||
/* do monitored operation */
|
||||
if(rank_world % 2) { /* Odd ranks : Recv - Send */
|
||||
MPI_Recv(rbuf, size, MPI_BYTE, from, tagno, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
|
||||
MPI_Send(sbuf, size, MPI_BYTE, from, tagno, MPI_COMM_WORLD);
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
get_tick(&start);
|
||||
MPI_Send(sbuf, size, MPI_BYTE, from, tagno, MPI_COMM_WORLD);
|
||||
MPI_Recv(rbuf, size, MPI_BYTE, from, tagno, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
|
||||
get_tick(&end);
|
||||
} else { /* Even ranks : Send - Recv */
|
||||
get_tick(&start);
|
||||
MPI_Send(sbuf, size, MPI_BYTE, to, tagno, MPI_COMM_WORLD);
|
||||
MPI_Recv(rbuf, size, MPI_BYTE, to, tagno, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
|
||||
get_tick(&end);
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
MPI_Recv(rbuf, size, MPI_BYTE, to, tagno, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
|
||||
MPI_Send(sbuf, size, MPI_BYTE, to, tagno, MPI_COMM_WORLD);
|
||||
}
|
||||
|
||||
*res = timing_delay(&start, &end) / 2;
|
||||
}
|
||||
|
||||
static inline void op_coll(double*res, void*buff, int size, int tagno, void*rbuf) {
|
||||
struct timespec start, end;
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
|
||||
/* do monitored operation */
|
||||
get_tick(&start);
|
||||
MPI_Bcast(buff, size, MPI_BYTE, 0, MPI_COMM_WORLD);
|
||||
get_tick(&end);
|
||||
|
||||
*res = timing_delay(&start, &end);
|
||||
}
|
||||
|
||||
static inline void op_a2a(double*res, void*sbuf, int size, int tagno, void*rbuf) {
|
||||
struct timespec start, end;
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
|
||||
/* do monitored operation */
|
||||
get_tick(&start);
|
||||
MPI_Alltoall(sbuf, size, MPI_BYTE, rbuf, size, MPI_BYTE, MPI_COMM_WORLD);
|
||||
get_tick(&end);
|
||||
|
||||
*res = timing_delay(&start, &end);
|
||||
}
|
||||
|
||||
static inline void op_put(double*res, void*sbuf, int size, int tagno, void*rbuf) {
|
||||
struct timespec start, end;
|
||||
|
||||
MPI_Win_lock(MPI_LOCK_EXCLUSIVE, to, 0, win);
|
||||
|
||||
/* do monitored operation */
|
||||
get_tick(&start);
|
||||
MPI_Put(sbuf, size, MPI_BYTE, to, 0, size, MPI_BYTE, win);
|
||||
MPI_Win_unlock(to, win);
|
||||
get_tick(&end);
|
||||
|
||||
*res = timing_delay(&start, &end);
|
||||
}
|
||||
|
||||
static inline void op_get(double*res, void*rbuf, int size, int tagno, void*sbuf) {
|
||||
struct timespec start, end;
|
||||
|
||||
MPI_Win_lock(MPI_LOCK_SHARED, to, 0, win);
|
||||
|
||||
/* do monitored operation */
|
||||
get_tick(&start);
|
||||
MPI_Get(rbuf, size, MPI_BYTE, to, 0, size, MPI_BYTE, win);
|
||||
MPI_Win_unlock(to, win);
|
||||
get_tick(&end);
|
||||
|
||||
*res = timing_delay(&start, &end);
|
||||
}
|
||||
|
||||
static inline void do_bench(int size, char*sbuf, double*results,
|
||||
void(*op)(double*, void*, int, int, void*)) {
|
||||
int iter;
|
||||
int tagno = 201;
|
||||
char*rbuf = sbuf ? sbuf + size : NULL;
|
||||
|
||||
if(op == op_put || op == op_get){
|
||||
win = MPI_WIN_NULL;
|
||||
MPI_Win_create(rbuf, size, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);
|
||||
}
|
||||
|
||||
for( iter = 0; iter < NB_ITER; ++iter ) {
|
||||
op(&results[iter], sbuf, size, tagno, rbuf);
|
||||
MPI_Barrier(MPI_COMM_WORLD);
|
||||
}
|
||||
|
||||
if(op == op_put || op == op_get){
|
||||
MPI_Win_free(&win);
|
||||
win = MPI_WIN_NULL;
|
||||
}
|
||||
}
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
int size, iter, nop;
|
||||
char*sbuf = NULL;
|
||||
double results[NB_ITER];
|
||||
void(*op)(double*, void*, int, int, void*);
|
||||
char name[255];
|
||||
MPI_Init(&argc, &argv);
|
||||
MPI_Comm_rank(MPI_COMM_WORLD, &rank_world);
|
||||
MPI_Comm_size(MPI_COMM_WORLD, &size_world);
|
||||
to = (rank_world + 1) % size_world;
|
||||
from = (rank_world + size_world - 1) % size_world;
|
||||
|
||||
double full_res[FULL_NB_ITER];
|
||||
|
||||
for( nop = 0; nop < NB_OPS; ++nop ) {
|
||||
switch(nop) {
|
||||
case 0:
|
||||
op = op_send;
|
||||
sprintf(name, "MPI_Send");
|
||||
break;
|
||||
case 1:
|
||||
op = op_coll;
|
||||
sprintf(name, "MPI_Bcast");
|
||||
break;
|
||||
case 2:
|
||||
op = op_a2a;
|
||||
sprintf(name, "MPI_Alltoall");
|
||||
break;
|
||||
case 3:
|
||||
op = op_put;
|
||||
sprintf(name, "MPI_Put");
|
||||
break;
|
||||
case 4:
|
||||
op = op_get;
|
||||
sprintf(name, "MPI_Get");
|
||||
break;
|
||||
case 5:
|
||||
op = op_send_pingpong;
|
||||
sprintf(name, "MPI_Send_pp");
|
||||
break;
|
||||
}
|
||||
|
||||
if( 0 == rank_world )
|
||||
printf("# %s%%%d\n# size \t| latency \t| 10^6 B/s \t| MB/s \t| median \t| q1 \t| q3 \t| d1 \t| d9 \t| avg \t| max\n", name, size_world);
|
||||
|
||||
for(size = 0; size < MAX_SIZE; size = ((int)(size * 1.4) > size) ? (size * 1.4) : (size + 1)) {
|
||||
/* Init buffers */
|
||||
if( 0 != size ) {
|
||||
sbuf = (char *)realloc(sbuf, (size_world + 1) * size); /* sbuf + alltoall recv buf */
|
||||
}
|
||||
|
||||
do_bench(size, sbuf, results, op);
|
||||
|
||||
MPI_Gather(results, NB_ITER, MPI_DOUBLE, full_res, NB_ITER, MPI_DOUBLE, 0, MPI_COMM_WORLD);
|
||||
|
||||
if( 0 == rank_world ) {
|
||||
qsort(full_res, FULL_NB_ITER, sizeof(double), &comp_double);
|
||||
const double min_lat = full_res[0];
|
||||
const double max_lat = full_res[FULL_NB_ITER - 1];
|
||||
const double med_lat = full_res[(FULL_NB_ITER - 1) / 2];
|
||||
const double q1_lat = full_res[(FULL_NB_ITER - 1) / 4];
|
||||
const double q3_lat = full_res[ 3 * (FULL_NB_ITER - 1) / 4];
|
||||
const double d1_lat = full_res[(FULL_NB_ITER - 1) / 10];
|
||||
const double d9_lat = full_res[ 9 * (FULL_NB_ITER - 1) / 10];
|
||||
double avg_lat = 0.0;
|
||||
for( iter = 0; iter < FULL_NB_ITER; iter++ ){
|
||||
avg_lat += full_res[iter];
|
||||
}
|
||||
avg_lat /= FULL_NB_ITER;
|
||||
const double bw_million_byte = size / min_lat;
|
||||
const double bw_mbyte = bw_million_byte / 1.048576;
|
||||
|
||||
printf("%9lld\t%9.3lf\t%9.3f\t%9.3f\t%9.3lf\t%9.3lf\t%9.3lf\t%9.3lf\t%9.3lf\t%9.3lf\t%9.3lf",
|
||||
(long long)size, min_lat, bw_million_byte, bw_mbyte,
|
||||
med_lat, q1_lat, q3_lat, d1_lat, d9_lat,
|
||||
avg_lat, max_lat);
|
||||
printf("\n");
|
||||
}
|
||||
}
|
||||
free(sbuf);
|
||||
sbuf = NULL;
|
||||
}
|
||||
|
||||
MPI_Finalize();
|
||||
return EXIT_SUCCESS;
|
||||
}
|
||||
216
test/monitoring/test_overhead.sh
Исполняемый файл
216
test/monitoring/test_overhead.sh
Исполняемый файл
@ -0,0 +1,216 @@
|
||||
#!/bin/bash
|
||||
|
||||
#
|
||||
# Copyright (c) 2016-2017 Inria. All rights reserved.
|
||||
# $COPYRIGHT$
|
||||
#
|
||||
# Additional copyrights may follow
|
||||
#
|
||||
# $HEADER$
|
||||
#
|
||||
|
||||
#
|
||||
# Author Clément Foyer <clement.foyer@inria.fr>
|
||||
#
|
||||
# This script launches the test_overhead test case for 2, 4, 8, 12,
|
||||
# 16, 20 and 24 processes, once with the monitoring component enabled,
|
||||
# and once without any monitoring. It then parses and aggregates the
|
||||
# results in order to create heatmaps. To work properly, this scripts
|
||||
# needs sqlite3, sed, awk and gnuplot. It also needs the rights to
|
||||
# write/create directories in the working path. Temporary files can be
|
||||
# found in $resdir/.tmp. They are cleaned between two executions fo
|
||||
# this script.
|
||||
#
|
||||
# This file create as an output one heatmap per operation
|
||||
# tested. Currently, tested operations are :
|
||||
# - MPI_Send (software overhead)
|
||||
# - MPI_Send (ping-pong, to measure theoverhead with the communciation time)
|
||||
# - MPI_Bcast
|
||||
# - MPI_Alltoall
|
||||
# - MPI_Put
|
||||
# - MPI_Get
|
||||
#
|
||||
|
||||
exe=test_overhead
|
||||
|
||||
# add common options
|
||||
if [ $# -ge 1 ]
|
||||
then
|
||||
mfile="-machinefile $1"
|
||||
fi
|
||||
common_opt="$mfile --bind-to core"
|
||||
|
||||
# dir
|
||||
resdir=res
|
||||
tmpdir=$resdir/.tmp
|
||||
# files
|
||||
base_nomon=$resdir/unmonitored
|
||||
base_mon=$resdir/monitored
|
||||
dbfile=$tmpdir/base.db
|
||||
dbscript=$tmpdir/overhead.sql
|
||||
plotfile=$tmpdir/plot.gp
|
||||
# operations
|
||||
ops=(send a2a bcast put get sendpp)
|
||||
|
||||
# no_monitoring(nb_nodes, exe_name, output_filename, error_filename)
|
||||
function no_monitoring() {
|
||||
mpiexec -n $1 $common_opt --mca pml ^monitoring --mca osc ^monitoring --mca coll ^monitoring $2 2> $4 > $3
|
||||
}
|
||||
|
||||
# monitoring(nb_nodes, exe_name, output_filename, error_filename)
|
||||
function monitoring() {
|
||||
mpiexec -n $1 $common_opt --mca pml_monitoring_enable 1 --mca pml_monitoring_enable_output 3 --mca pml_monitoring_filename "prof/toto" $2 2> $4 > $3
|
||||
}
|
||||
|
||||
# filter_output(filenames_list)
|
||||
function filter_output() {
|
||||
for filename in "$@"
|
||||
do
|
||||
# remove extra texts from the output
|
||||
sed -i '/--------------------------------------------------------------------------/,/--------------------------------------------------------------------------/d' $filename
|
||||
# create all sub files as $tmpdir/$filename
|
||||
file=$(sed -e "s|$resdir/|$tmpdir/|" -e "s/\.dat/.csv/" <<< $filename)
|
||||
# split in file, one per kind of operation monitored
|
||||
awk "/^# MPI_Send/ {out=\"$(sed "s/\.$nbprocs/.send&/" <<< $file)\"}; \
|
||||
/^# MPI_Bcast/ {out=\"$(sed "s/\.$nbprocs/.bcast&/" <<< $file)\"}; \
|
||||
/^# MPI_Alltoall/ {out=\"$(sed "s/\.$nbprocs/.a2a&/" <<< $file)\"}; \
|
||||
/^# MPI_Put/ {out=\"$(sed "s/\.$nbprocs/.put&/" <<< $file)\"}; \
|
||||
/^# MPI_Get/ {out=\"$(sed "s/\.$nbprocs/.get&/" <<< $file)\"}; \
|
||||
/^# MPI_Send_pp/ {out=\"$(sed "s/\.$nbprocs/.sendpp&/" <<< $file)\"}; \
|
||||
/^#/ { } ; !/^#/ {\$0=\"$nbprocs \"\$0; print > out};" \
|
||||
out=$tmpdir/tmp $filename
|
||||
done
|
||||
# trim spaces and replace them with comma in each file generated with awk
|
||||
for file in `ls $tmpdir/*.*.$nbprocs.csv`
|
||||
do
|
||||
sed -i 's/[[:space:]]\{1,\}/,/g' $file
|
||||
done
|
||||
}
|
||||
|
||||
# clean previous execution if any
|
||||
if [ -d $tmpdir ]
|
||||
then
|
||||
rm -fr $tmpdir
|
||||
fi
|
||||
mkdir -p $tmpdir
|
||||
|
||||
# start creating the sql file for data post-processing
|
||||
cat > $dbscript <<EOF
|
||||
-- Enables mode Comma-Separated Values for input and output
|
||||
.mode csv
|
||||
|
||||
-- Optionally enables column header display on output
|
||||
.header off
|
||||
|
||||
-- Create one empty table for each of the input CSV files
|
||||
EOF
|
||||
|
||||
for op in ${ops[*]}
|
||||
do
|
||||
echo -e "drop table if exists ${op}_mon;\ndrop table if exists ${op}_nomon;" >> $dbscript
|
||||
echo -e "create table if not exists ${op}_mon (nbprocs integer, datasize integer, lat float, speed float, MBspeed float, media float, q1 float, q3 float, d1 float, d9 float, average float, maximum float, primary key (nbprocs, datasize) on conflict abort);\ncreate table if not exists ${op}_nomon (nbprocs integer, datasize integer, lat float, speed float, MBspeed float, media float, q1 float, q3 float, d1 float, d9 float, average float, maximum float, primary key (nbprocs, datasize) on conflict abort);" >> $dbscript
|
||||
done
|
||||
|
||||
# main loop to launch benchmarks
|
||||
for nbprocs in 2 4 8 12 16 20 24
|
||||
do
|
||||
echo "$nbprocs procs..."
|
||||
output_nomon="$base_nomon.$nbprocs.dat"
|
||||
error_nomon="$base_nomon.$nbprocs.err"
|
||||
output_mon="$base_mon.$nbprocs.dat"
|
||||
error_mon="$base_mon.$nbprocs.err"
|
||||
# actually do the benchmarks
|
||||
no_monitoring $nbprocs $exe $output_nomon $error_nomon
|
||||
monitoring $nbprocs $exe $output_mon $error_mon
|
||||
# prepare data to insert them more easily into database
|
||||
filter_output $output_nomon $output_mon
|
||||
# insert into database
|
||||
echo -e "\n-- Import each CSV file in its corresponding table" >> $dbscript
|
||||
for op in ${ops[*]}
|
||||
do
|
||||
echo -e ".import $(sed "s|$resdir/|$tmpdir/|" <<<$base_mon).${op}.${nbprocs}.csv ${op}_mon\n.import $(sed "s|$resdir/|$tmpdir/|" <<<$base_nomon).${op}.${nbprocs}.csv ${op}_nomon" >> $dbscript
|
||||
done
|
||||
done
|
||||
|
||||
echo "Fetch data..."
|
||||
echo -e "\n-- Perform some select query" >> $dbscript
|
||||
for op in ${ops[*]}
|
||||
do
|
||||
cat >> $dbscript <<EOF
|
||||
-- set file output
|
||||
.output $tmpdir/${op}.dat
|
||||
-- do query for overheads
|
||||
select ${op}_mon.nbprocs as nbprocs, ${op}_mon.datasize as datasize, ${op}_mon.media/${op}_nomon.media-1
|
||||
from ${op}_mon inner join ${op}_nomon on (${op}_mon.nbprocs==${op}_nomon.nbprocs and ${op}_mon.datasize==${op}_nomon.datasize)
|
||||
order by nbprocs, datasize;
|
||||
|
||||
EOF
|
||||
done
|
||||
cat >> $dbscript <<EOF
|
||||
-- create view for all overheads
|
||||
create temporary view medians as
|
||||
select NULL as ovh
|
||||
EOF
|
||||
for op in ${ops[*]}
|
||||
do
|
||||
cat >> $dbscript <<EOF
|
||||
union all select all ${op}_mon.media / ${op}_nomon.media - 1 as ovh from ${op}_mon inner join ${op}_nomon on (${op}_mon.nbprocs == ${op}_nomon.nbprocs and ${op}_mon.datasize == ${op}_nomon.datasize)
|
||||
EOF
|
||||
done
|
||||
cat >> $dbscript <<EOF
|
||||
;
|
||||
|
||||
select '# average of overheads: ', avg(ovh) from medians where ovh is not null;
|
||||
select '# median of overheads: ', avg(ovh) from (
|
||||
select ovh from medians
|
||||
where ovh is not null
|
||||
order by ovh
|
||||
limit 2 - (select count(*) from medians where ovh is not null) % 2
|
||||
offset(select (count(*) - 1) / 2 from medians where ovh is not null)
|
||||
);
|
||||
|
||||
-- End of the script
|
||||
.quit
|
||||
EOF
|
||||
|
||||
# data post processing + create output files
|
||||
sqlite3 $dbfile < $dbscript
|
||||
|
||||
# create plotting script
|
||||
cat > $plotfile <<EOF
|
||||
set terminal pngcairo
|
||||
#set key inside bottom right box
|
||||
set key bmargin box
|
||||
set key off
|
||||
#set grid
|
||||
set view map
|
||||
set pm3d
|
||||
#set origin 2,0
|
||||
|
||||
set xlabel "Nb procs"
|
||||
set ylabel "Message size (bytes)"
|
||||
set cblabel "Monitoring overhead (overhead ratio)"
|
||||
set cbtics format '%.1f'
|
||||
|
||||
set ytics ( "4 B" 4, "16 B" 16, "128 B" 128, "1 KB" 1024, "8 KB" 8*1024, "128 KB" 128*1024, "1 MB" 1024*1024 )
|
||||
set xtics ( 2, 4, 8, 12, 16, 20, 24 )
|
||||
set xrange [2:24]
|
||||
#set cbrange [0:1]
|
||||
set nologscale x
|
||||
set logscale y 2
|
||||
|
||||
$(
|
||||
for op in ${ops[*]}
|
||||
do
|
||||
# finish to prepare the data files
|
||||
sed -i -e "s/,/ /g" $tmpdir/${op}.dat
|
||||
awk -F ' ' -v val=0 '{ if (val<$1) { val=$1 ; print " " > out ; print $0 > out } else { print $0 > out } }' out=$tmpdir/${op}.dat $tmpdir/${op}.dat
|
||||
echo -e "set output '$resdir/${op}.png'\nsplot '$tmpdir/${op}.dat' using (\$1):(\$2):(\$3) with pm3d"
|
||||
done)
|
||||
EOF
|
||||
|
||||
echo "Generating graphs..."
|
||||
|
||||
gnuplot < $plotfile
|
||||
|
||||
echo "Done."
|
||||
323
test/monitoring/test_pvar_access.c
Обычный файл
323
test/monitoring/test_pvar_access.c
Обычный файл
@ -0,0 +1,323 @@
|
||||
/*
|
||||
* Copyright (c) 2013-2017 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2013-2016 Inria. All rights reserved.
|
||||
* Copyright (c) 2015 Cisco Systems, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
/*
|
||||
pml monitoring tester.
|
||||
|
||||
Designed by George Bosilca <bosilca@icl.utk.edu>, Emmanuel Jeannot <emmanuel.jeannot@inria.fr> and
|
||||
Clement Foyer <clement.foyer@inria.fr>
|
||||
Contact the authors for questions.
|
||||
|
||||
To be run as:
|
||||
|
||||
mpirun -np 4 --mca pml_monitoring_enable 2 ./test_pvar_access
|
||||
|
||||
Then, the output should be:
|
||||
Flushing phase 1:
|
||||
I 0 1 108 bytes 27 msgs sent
|
||||
I 1 2 104 bytes 26 msgs sent
|
||||
I 2 3 104 bytes 26 msgs sent
|
||||
I 3 0 104 bytes 26 msgs sent
|
||||
Flushing phase 2:
|
||||
I 0 1 20 bytes 4 msgs sent
|
||||
I 0 2 20528 bytes 9 msgs sent
|
||||
I 1 0 20 bytes 4 msgs sent
|
||||
I 1 2 104 bytes 26 msgs sent
|
||||
I 1 3 236 bytes 56 msgs sent
|
||||
I 2 0 20528 bytes 9 msgs sent
|
||||
I 2 3 112 bytes 27 msgs sent
|
||||
I 3 1 220 bytes 52 msgs sent
|
||||
I 3 2 20 bytes 4 msgs sent
|
||||
|
||||
*/
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <mpi.h>
|
||||
|
||||
static MPI_T_pvar_handle count_handle;
|
||||
static MPI_T_pvar_handle msize_handle;
|
||||
static const char count_pvar_name[] = "pml_monitoring_messages_count";
|
||||
static const char msize_pvar_name[] = "pml_monitoring_messages_size";
|
||||
static int count_pvar_idx, msize_pvar_idx;
|
||||
static int world_rank, world_size;
|
||||
|
||||
static void print_vars(int rank, int size, size_t* msg_count, size_t*msg_size)
|
||||
{
|
||||
int i;
|
||||
for(i = 0; i < size; ++i) {
|
||||
if(0 != msg_size[i])
|
||||
printf("I\t%d\t%d\t%zu bytes\t%zu msgs sent\n", rank, i, msg_size[i], msg_count[i]);
|
||||
}
|
||||
}
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
int rank, size, n, to, from, tagno, MPIT_result, provided, count;
|
||||
MPI_T_pvar_session session;
|
||||
MPI_Status status;
|
||||
MPI_Comm newcomm;
|
||||
MPI_Request request;
|
||||
size_t*msg_count_p1, *msg_size_p1;
|
||||
size_t*msg_count_p2, *msg_size_p2;
|
||||
|
||||
/* first phase : make a token circulated in MPI_COMM_WORLD */
|
||||
n = -1;
|
||||
MPI_Init(&argc, &argv);
|
||||
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
|
||||
MPI_Comm_size(MPI_COMM_WORLD, &size);
|
||||
world_size = size;
|
||||
world_rank = rank;
|
||||
to = (rank + 1) % size;
|
||||
from = (rank - 1) % size;
|
||||
tagno = 201;
|
||||
|
||||
MPIT_result = MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);
|
||||
if (MPIT_result != MPI_SUCCESS)
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
|
||||
/* Retrieve the pvar indices */
|
||||
MPIT_result = MPI_T_pvar_get_index(count_pvar_name, MPI_T_PVAR_CLASS_SIZE, &count_pvar_idx);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot find monitoring MPI_T \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_get_index(msize_pvar_name, MPI_T_PVAR_CLASS_SIZE, &msize_pvar_idx);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot find monitoring MPI_T \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Get session for pvar binding */
|
||||
MPIT_result = MPI_T_pvar_session_create(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot create a session for \"%s\" and \"%s\" pvars\n",
|
||||
count_pvar_name, msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Allocating a new PVAR in a session will reset the counters */
|
||||
MPIT_result = MPI_T_pvar_handle_alloc(session, count_pvar_idx,
|
||||
MPI_COMM_WORLD, &count_handle, &count);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to allocate handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_handle_alloc(session, msize_pvar_idx,
|
||||
MPI_COMM_WORLD, &msize_handle, &count);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to allocate handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Allocate arrays to retrieve results */
|
||||
msg_count_p1 = calloc(count * 4, sizeof(size_t));
|
||||
msg_size_p1 = &msg_count_p1[count];
|
||||
msg_count_p2 = &msg_count_p1[2*count];
|
||||
msg_size_p2 = &msg_count_p1[3*count];
|
||||
|
||||
/* Start pvar */
|
||||
MPIT_result = MPI_T_pvar_start(session, count_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_start(session, msize_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
if (rank == 0) {
|
||||
n = 25;
|
||||
MPI_Isend(&n,1,MPI_INT,to,tagno,MPI_COMM_WORLD,&request);
|
||||
}
|
||||
while (1) {
|
||||
MPI_Irecv(&n, 1, MPI_INT, from, tagno, MPI_COMM_WORLD, &request);
|
||||
MPI_Wait(&request, &status);
|
||||
if (rank == 0) {n--;tagno++;}
|
||||
MPI_Isend(&n, 1, MPI_INT, to, tagno, MPI_COMM_WORLD, &request);
|
||||
if (rank != 0) {n--;tagno++;}
|
||||
if (n<0){
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/* Test stopping variable then get values */
|
||||
MPIT_result = MPI_T_pvar_stop(session, count_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_stop(session, msize_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to stop handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_read(session, count_handle, msg_count_p1);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to fetch handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_read(session, msize_handle, msg_size_p1);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to fetch handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Circulate a token to proper display the results */
|
||||
if(0 == world_rank) {
|
||||
printf("Flushing phase 1:\n");
|
||||
print_vars(world_rank, world_size, msg_count_p1, msg_size_p1);
|
||||
MPI_Send(NULL, 0, MPI_BYTE, (world_rank + 1) % world_size, 300, MPI_COMM_WORLD);
|
||||
MPI_Recv(NULL, 0, MPI_BYTE, (world_rank - 1) % world_size, 300, MPI_COMM_WORLD, &status);
|
||||
} else {
|
||||
MPI_Recv(NULL, 0, MPI_BYTE, (world_rank - 1) % world_size, 300, MPI_COMM_WORLD, &status);
|
||||
print_vars(world_rank, world_size, msg_count_p1, msg_size_p1);
|
||||
MPI_Send(NULL, 0, MPI_BYTE, (world_rank + 1) % world_size, 300, MPI_COMM_WORLD);
|
||||
}
|
||||
|
||||
/* Add to the phase 1 the display token ring message count */
|
||||
MPIT_result = MPI_T_pvar_read(session, count_handle, msg_count_p1);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to fetch handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_read(session, msize_handle, msg_size_p1);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to fetch handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/*
|
||||
Second phase. Work with different communicators.
|
||||
even ranks will circulate a token
|
||||
while odd ranks will perform a all_to_all
|
||||
*/
|
||||
MPIT_result = MPI_T_pvar_start(session, count_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_start(session, msize_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to start handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPI_Comm_split(MPI_COMM_WORLD, rank%2, rank, &newcomm);
|
||||
|
||||
if(rank%2){ /*even ranks (in COMM_WORD) circulate a token*/
|
||||
MPI_Comm_rank(newcomm, &rank);
|
||||
MPI_Comm_size(newcomm, &size);
|
||||
if( size > 1 ) {
|
||||
to = (rank + 1) % size;
|
||||
from = (rank - 1) % size;
|
||||
tagno = 201;
|
||||
if (rank == 0){
|
||||
n = 50;
|
||||
MPI_Send(&n, 1, MPI_INT, to, tagno, newcomm);
|
||||
}
|
||||
while (1){
|
||||
MPI_Recv(&n, 1, MPI_INT, from, tagno, newcomm, &status);
|
||||
if (rank == 0) {n--; tagno++;}
|
||||
MPI_Send(&n, 1, MPI_INT, to, tagno, newcomm);
|
||||
if (rank != 0) {n--; tagno++;}
|
||||
if (n<0){
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
} else { /*odd ranks (in COMM_WORD) will perform a all_to_all and a barrier*/
|
||||
int send_buff[10240];
|
||||
int recv_buff[10240];
|
||||
MPI_Comm_rank(newcomm, &rank);
|
||||
MPI_Comm_size(newcomm, &size);
|
||||
MPI_Alltoall(send_buff, 10240/size, MPI_INT, recv_buff, 10240/size, MPI_INT, newcomm);
|
||||
MPI_Comm_split(newcomm, rank%2, rank, &newcomm);
|
||||
MPI_Barrier(newcomm);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_read(session, count_handle, msg_count_p2);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to fetch handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_read(session, msize_handle, msg_size_p2);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to fetch handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
/* Taking only in account the second phase */
|
||||
for(int i = 0; i < size; ++i) {
|
||||
msg_count_p2[i] -= msg_count_p1[i];
|
||||
msg_size_p2[i] -= msg_size_p1[i];
|
||||
}
|
||||
|
||||
/* Circulate a token to proper display the results */
|
||||
if(0 == world_rank) {
|
||||
printf("Flushing phase 2:\n");
|
||||
print_vars(world_rank, world_size, msg_count_p2, msg_size_p2);
|
||||
MPI_Send(NULL, 0, MPI_BYTE, (world_rank + 1) % world_size, 300, MPI_COMM_WORLD);
|
||||
MPI_Recv(NULL, 0, MPI_BYTE, (world_rank - 1) % world_size, 300, MPI_COMM_WORLD, &status);
|
||||
} else {
|
||||
MPI_Recv(NULL, 0, MPI_BYTE, (world_rank - 1) % world_size, 300, MPI_COMM_WORLD, &status);
|
||||
print_vars(world_rank, world_size, msg_count_p2, msg_size_p2);
|
||||
MPI_Send(NULL, 0, MPI_BYTE, (world_rank + 1) % world_size, 300, MPI_COMM_WORLD);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_handle_free(session, &count_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to free handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
count_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
MPIT_result = MPI_T_pvar_handle_free(session, &msize_handle);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("failed to free handle on \"%s\" pvar, check that you have monitoring pml\n",
|
||||
msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
MPIT_result = MPI_T_pvar_session_free(&session);
|
||||
if (MPIT_result != MPI_SUCCESS) {
|
||||
printf("cannot close a session for \"%s\" and \"%s\" pvars\n",
|
||||
count_pvar_name, msize_pvar_name);
|
||||
MPI_Abort(MPI_COMM_WORLD, MPIT_result);
|
||||
}
|
||||
|
||||
(void)MPI_T_finalize();
|
||||
|
||||
free(msg_count_p1);
|
||||
|
||||
MPI_Finalize();
|
||||
return EXIT_SUCCESS;
|
||||
}
|
||||
Загрузка…
x
Ссылка в новой задаче
Block a user