
Add a monitoring PML, OSC and IO. They track all data exchanges between processes, with capability to include or exclude collective traffic. The monitoring infrastructure is driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows. Documentations and examples have been added, as well as a shared library that can be used with LD_PRELOAD and that allows the monitoring of any application. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * add ability to querry pml monitorinting results with MPI Tools interface using performance variables "pml_monitoring_messages_count" and "pml_monitoring_messages_size" Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Fix a convertion problem and add a comment about the lack of component retain in the new component infrastructure. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Allow the pvar to be written by invoking the associated callback. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Various fixes for the monitoring. Allocate all counting arrays in a single allocation Don't delay the initialization (do it at the first add_proc as we know the number of processes in MPI_COMM_WORLD) Add a choice: with or without MPI_T (default). Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Cleanup for the monitoring module. Fixed few bugs, and reshape the operations to prepare for global or communicator-based monitoring. Start integrating support for MPI_T as well as MCA monitoring. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Adding documentation about how to use pml_monitoring component. Document present the use with and without MPI_T. May not reflect exactly how it works right now, but should reflects how it should work in the end. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c. Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Improve monitoring support (including integration with MPI_T) Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output. Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example Set filename only if using mpi tools Adding missing parameters for fprintf in monitoring_flush (for output in std's cases) Fix expected output/results for example header Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer Base whether to output or not on message count, in order to print something if only empty messages are exchanged Add a new example on how to access performance variables from within the code Allocate arrays regarding value returned by binding Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add overhead benchmark, with script to use data and create graphs out of the results Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix segfault error at end when not loading pml Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Start create common monitoring module. Factorise version numbering Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix microbenchmarks script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Improve readability of code NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add osc monitoring component Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error checking if running out of memory in osc_monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Resolve brutal segfault when double freeing filename Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Moving to ompi/mca/common the proper parts of the monitoring system Using common functions instead of pml specific one. Removing pml ones. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add calls to record monitored data from osc. Use common function to translate ranks. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix test_overhead benchmark script distribution Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix linking library with mca/common Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add passive operations in monitoring_test Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix from rank calculation. Add more detailed error messages Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix osc_monitoring mget_message_count function call Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring common output system Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Consistent output file name (with and without MPI_T). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Always output to a file when flushing at pvar_stop(flush). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update the monitoring documentation. Complete informations from HowTo. Fix a few mistake and typos. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use the world_rank for printf's. Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add security check for unique initialization for osc monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the amout of symbols available outside mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Remove use of __sync_* built-ins. Use opal_atomic_* instead. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Deleting now useless file : moved to common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram ditribution of message sizes Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add coll component for collectives communications monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix log10_2 constant initialization. Fix index calculation for histogram array. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add debug info messages to follow more easily initialization steps. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring. monitoring_filter only indicates if filtering is activated. Fix out of range access in histogram. List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t. Remove useless dead code. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Don't install the test scripts. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix missing procs in hashtable. Cache coll monitoring data. * Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer. * Cache monitoring data relative to collectives operations on creation. * Remove double caching. * Use same proc name definition for hash table when inserting and when retrieving. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use intermediate variable to avoid invalid write while retrieving ranks in hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add missing release of the last element in flush_all. Add release of the hashtable in finalize. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Set world_rank from hashtable only if found Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use predefined symbol from opal system to print int Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add automated check (with MPI_Tools) of monitoring. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix procs list caching in common_monitoring_coll_data_t * Fix monitoring_coll_data type definition. * Use size(COMM_WORLD)-1 to determine max number of digits. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Documentation update. Update and then move the latex and README documentation to a more logical place Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add the use of a machine file for overhead benchmark Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Check for out-of-bound write in histogram Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix common_monitoring_cache object init for MPI_COMM_WORLD Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add RDMA benchmarks to test_overhead Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add technical documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adapt to the new definition of communicators Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update expected output in test/monitoring/monitoring_test.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add dumping histogram in edge case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add consistency in header inclusion. Include ompi/mpi/fortran/mpif-h/bindings.h only if needed. Add sanity check before emptying hashtable. Fix typos in documentation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * misc monitoring fixes * test/monitoring: fix test when weak symbols are not available * monitoring: fix a typo and add a missing file in Makefile.am and have monitoring_common.h and monitoring_common_coll.h included in the distro * test/monitoring: cleanup all tests and make distclean a happy panda * test/monitoring: use gettimeofday() if clock_gettime() is unavailable * monitoring: silence misc warnings (#3) Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp> * Cleanups. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Changing int64_t to size_t. Keep the size_t used accross all monitoring components. Adapt the documentation. Remove useless MPI_Request and MPI_Status from monitoring_test.c. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add parameter for RMA test case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the maximum bound computation for proc list dump. Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add communicator-specific monitored collective data reset Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring scripts to the 'make dist' Also install them in the build and the install directories. Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
160 строки
6.0 KiB
C
160 строки
6.0 KiB
C
/*
|
|
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
|
* University Research and Technology
|
|
* Corporation. All rights reserved.
|
|
* Copyright (c) 2004-2017 The University of Tennessee and The University
|
|
* of Tennessee Research Foundation. All rights
|
|
* reserved.
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
* University of Stuttgart. All rights reserved.
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
* All rights reserved.
|
|
* Copyright (c) 2008-2014 Cisco Systems, Inc. All rights reserved.
|
|
* Copyright (c) 2012 Los Alamos National Security, LLC.
|
|
* All rights reserved.
|
|
* Copyright (c) 2014 Research Organization for Information Science
|
|
* and Technology (RIST). All rights reserved.
|
|
* $COPYRIGHT$
|
|
*
|
|
* Additional copyrights may follow
|
|
*
|
|
* $HEADER$
|
|
*/
|
|
|
|
#include "ompi_config.h"
|
|
|
|
#include <stdio.h>
|
|
#include <stdlib.h>
|
|
#include <string.h>
|
|
|
|
#include "mpi.h"
|
|
#include "opal/util/show_help.h"
|
|
#include "ompi/constants.h"
|
|
#include "opal/class/opal_list.h"
|
|
#include "opal/util/output.h"
|
|
#include "ompi/mca/mca.h"
|
|
#include "opal/mca/base/base.h"
|
|
#include "opal/mca/base/mca_base_component_repository.h"
|
|
|
|
#include "ompi/mca/rte/rte.h"
|
|
#include "ompi/mca/coll/coll.h"
|
|
#include "ompi/mca/coll/base/base.h"
|
|
|
|
/*
|
|
* Private functions
|
|
*/
|
|
static int init_query(const mca_base_component_t * ls,
|
|
bool enable_progress_threads,
|
|
bool enable_mpi_threads);
|
|
|
|
/*
|
|
* Scan down the list of successfully opened components and query each of
|
|
* them (the opened list will be one or more components. If the user
|
|
* requested a specific component, it will be the only component in the
|
|
* opened list). Create and populate the available list of all
|
|
* components who indicate that they want to be considered for selection.
|
|
* Close all components who do not want to be considered for selection,
|
|
* and destroy the opened list.
|
|
*
|
|
* Also find the basic component while we're doing all of this, and save
|
|
* it in a global variable so that we can find it easily later (e.g.,
|
|
* during scope selection).
|
|
*/
|
|
int mca_coll_base_find_available(bool enable_progress_threads,
|
|
bool enable_mpi_threads)
|
|
{
|
|
mca_base_component_list_item_t *cli, *next;
|
|
const mca_base_component_t *component;
|
|
|
|
/* The list of components that we should check has already been
|
|
established in mca_coll_base_open. */
|
|
|
|
OPAL_LIST_FOREACH_SAFE(cli, next, &ompi_coll_base_framework.framework_components, mca_base_component_list_item_t) {
|
|
component = cli->cli_component;
|
|
|
|
/* Call a subroutine to do the work, because the component may
|
|
represent different versions of the coll MCA. */
|
|
|
|
if (OMPI_SUCCESS != init_query(component, enable_progress_threads,
|
|
enable_mpi_threads)) {
|
|
/* If the component doesn't want to run, then close it.
|
|
Now close it out and release it from the DSO repository (if it's there). */
|
|
opal_list_remove_item(&ompi_coll_base_framework.framework_components, &cli->super);
|
|
mca_base_component_close(component, ompi_coll_base_framework.framework_output);
|
|
OBJ_RELEASE(cli);
|
|
}
|
|
}
|
|
|
|
/* If we have no collective components available, it's an error.
|
|
Thanks for playing! */
|
|
|
|
if (opal_list_get_size(&ompi_coll_base_framework.framework_components) == 0) {
|
|
opal_output_verbose(10, ompi_coll_base_framework.framework_output,
|
|
"coll:find_available: no coll components available!");
|
|
opal_show_help("help-mca-base.txt", "find-available:not-valid", true,
|
|
"coll");
|
|
return OMPI_ERROR;
|
|
}
|
|
|
|
/* All done */
|
|
|
|
return OMPI_SUCCESS;
|
|
}
|
|
|
|
|
|
/*
|
|
* Query a specific component, coll v2.0.0
|
|
*/
|
|
static inline int
|
|
init_query_2_0_0(const mca_base_component_t * component,
|
|
bool enable_progress_threads,
|
|
bool enable_mpi_threads)
|
|
{
|
|
mca_coll_base_component_2_0_0_t *coll =
|
|
(mca_coll_base_component_2_0_0_t *) component;
|
|
|
|
return coll->collm_init_query(enable_progress_threads,
|
|
enable_mpi_threads);
|
|
}
|
|
/*
|
|
* Query a component, see if it wants to run at all. If it does, save
|
|
* some information. If it doesn't, close it.
|
|
*/
|
|
static int init_query(const mca_base_component_t * component,
|
|
bool enable_progress_threads, bool enable_mpi_threads)
|
|
{
|
|
int ret;
|
|
|
|
opal_output_verbose(10, ompi_coll_base_framework.framework_output,
|
|
"coll:find_available: querying coll component %s",
|
|
component->mca_component_name);
|
|
|
|
/* This component has already been successfully opened. So now
|
|
query it. */
|
|
|
|
if (2 == component->mca_type_major_version &&
|
|
0 == component->mca_type_minor_version &&
|
|
0 == component->mca_type_release_version) {
|
|
ret = init_query_2_0_0(component, enable_progress_threads,
|
|
enable_mpi_threads);
|
|
} else {
|
|
/* Unrecognized coll API version */
|
|
|
|
opal_output_verbose(10, ompi_coll_base_framework.framework_output,
|
|
"coll:find_available: unrecognized coll API version (%d.%d.%d, ignored)",
|
|
component->mca_type_major_version,
|
|
component->mca_type_minor_version,
|
|
component->mca_type_release_version);
|
|
return OMPI_ERROR;
|
|
}
|
|
|
|
/* Query done -- look at the return value to see what happened */
|
|
opal_output_verbose(10, ompi_coll_base_framework.framework_output,
|
|
"coll:find_available: coll component %s is %savailable",
|
|
component->mca_component_name,
|
|
(OMPI_SUCCESS == ret) ? "": "not ");
|
|
|
|
return ret;
|
|
}
|
|
|