1
1
openmpi/ompi/mca/osc/monitoring/configure.m4

101 строка
2.7 KiB
Plaintext
Исходник Обычный вид История

dnl -*- shell-script -*-
dnl
dnl Copyright (c) 2016-2018 Inria. All rights reserved.
dnl $COPYRIGHT$
dnl
dnl Additional copyrights may follow
dnl
dnl $HEADER$
dnl
# mca_ompi_osc_monitoring_generate_templates
Topic/monitoring (#3109) Add a monitoring PML, OSC and IO. They track all data exchanges between processes, with capability to include or exclude collective traffic. The monitoring infrastructure is driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows. Documentations and examples have been added, as well as a shared library that can be used with LD_PRELOAD and that allows the monitoring of any application. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * add ability to querry pml monitorinting results with MPI Tools interface using performance variables "pml_monitoring_messages_count" and "pml_monitoring_messages_size" Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Fix a convertion problem and add a comment about the lack of component retain in the new component infrastructure. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Allow the pvar to be written by invoking the associated callback. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Various fixes for the monitoring. Allocate all counting arrays in a single allocation Don't delay the initialization (do it at the first add_proc as we know the number of processes in MPI_COMM_WORLD) Add a choice: with or without MPI_T (default). Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Cleanup for the monitoring module. Fixed few bugs, and reshape the operations to prepare for global or communicator-based monitoring. Start integrating support for MPI_T as well as MCA monitoring. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Adding documentation about how to use pml_monitoring component. Document present the use with and without MPI_T. May not reflect exactly how it works right now, but should reflects how it should work in the end. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c. Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Improve monitoring support (including integration with MPI_T) Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output. Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example Set filename only if using mpi tools Adding missing parameters for fprintf in monitoring_flush (for output in std's cases) Fix expected output/results for example header Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer Base whether to output or not on message count, in order to print something if only empty messages are exchanged Add a new example on how to access performance variables from within the code Allocate arrays regarding value returned by binding Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add overhead benchmark, with script to use data and create graphs out of the results Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix segfault error at end when not loading pml Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Start create common monitoring module. Factorise version numbering Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix microbenchmarks script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Improve readability of code NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add osc monitoring component Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error checking if running out of memory in osc_monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Resolve brutal segfault when double freeing filename Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Moving to ompi/mca/common the proper parts of the monitoring system Using common functions instead of pml specific one. Removing pml ones. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add calls to record monitored data from osc. Use common function to translate ranks. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix test_overhead benchmark script distribution Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix linking library with mca/common Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add passive operations in monitoring_test Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix from rank calculation. Add more detailed error messages Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix osc_monitoring mget_message_count function call Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring common output system Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Consistent output file name (with and without MPI_T). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Always output to a file when flushing at pvar_stop(flush). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update the monitoring documentation. Complete informations from HowTo. Fix a few mistake and typos. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use the world_rank for printf's. Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add security check for unique initialization for osc monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the amout of symbols available outside mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Remove use of __sync_* built-ins. Use opal_atomic_* instead. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Deleting now useless file : moved to common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram ditribution of message sizes Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add coll component for collectives communications monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix log10_2 constant initialization. Fix index calculation for histogram array. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add debug info messages to follow more easily initialization steps. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring. monitoring_filter only indicates if filtering is activated. Fix out of range access in histogram. List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t. Remove useless dead code. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Don't install the test scripts. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix missing procs in hashtable. Cache coll monitoring data. * Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer. * Cache monitoring data relative to collectives operations on creation. * Remove double caching. * Use same proc name definition for hash table when inserting and when retrieving. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use intermediate variable to avoid invalid write while retrieving ranks in hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add missing release of the last element in flush_all. Add release of the hashtable in finalize. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Set world_rank from hashtable only if found Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use predefined symbol from opal system to print int Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add automated check (with MPI_Tools) of monitoring. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix procs list caching in common_monitoring_coll_data_t * Fix monitoring_coll_data type definition. * Use size(COMM_WORLD)-1 to determine max number of digits. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Documentation update. Update and then move the latex and README documentation to a more logical place Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add the use of a machine file for overhead benchmark Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Check for out-of-bound write in histogram Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix common_monitoring_cache object init for MPI_COMM_WORLD Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add RDMA benchmarks to test_overhead Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add technical documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adapt to the new definition of communicators Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update expected output in test/monitoring/monitoring_test.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add dumping histogram in edge case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add consistency in header inclusion. Include ompi/mpi/fortran/mpif-h/bindings.h only if needed. Add sanity check before emptying hashtable. Fix typos in documentation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * misc monitoring fixes * test/monitoring: fix test when weak symbols are not available * monitoring: fix a typo and add a missing file in Makefile.am and have monitoring_common.h and monitoring_common_coll.h included in the distro * test/monitoring: cleanup all tests and make distclean a happy panda * test/monitoring: use gettimeofday() if clock_gettime() is unavailable * monitoring: silence misc warnings (#3) Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp> * Cleanups. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Changing int64_t to size_t. Keep the size_t used accross all monitoring components. Adapt the documentation. Remove useless MPI_Request and MPI_Status from monitoring_test.c. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add parameter for RMA test case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the maximum bound computation for proc list dump. Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add communicator-specific monitored collective data reset Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring scripts to the 'make dist' Also install them in the build and the install directories. Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
2017-06-26 19:21:39 +03:00
#
# Overwrite $1. $1 is where the different templates are brought
# together and compose an array of components by listing component
# names in $2.
Topic/monitoring (#3109) Add a monitoring PML, OSC and IO. They track all data exchanges between processes, with capability to include or exclude collective traffic. The monitoring infrastructure is driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows. Documentations and examples have been added, as well as a shared library that can be used with LD_PRELOAD and that allows the monitoring of any application. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * add ability to querry pml monitorinting results with MPI Tools interface using performance variables "pml_monitoring_messages_count" and "pml_monitoring_messages_size" Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Fix a convertion problem and add a comment about the lack of component retain in the new component infrastructure. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Allow the pvar to be written by invoking the associated callback. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Various fixes for the monitoring. Allocate all counting arrays in a single allocation Don't delay the initialization (do it at the first add_proc as we know the number of processes in MPI_COMM_WORLD) Add a choice: with or without MPI_T (default). Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Cleanup for the monitoring module. Fixed few bugs, and reshape the operations to prepare for global or communicator-based monitoring. Start integrating support for MPI_T as well as MCA monitoring. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Adding documentation about how to use pml_monitoring component. Document present the use with and without MPI_T. May not reflect exactly how it works right now, but should reflects how it should work in the end. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c. Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Improve monitoring support (including integration with MPI_T) Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output. Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example Set filename only if using mpi tools Adding missing parameters for fprintf in monitoring_flush (for output in std's cases) Fix expected output/results for example header Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer Base whether to output or not on message count, in order to print something if only empty messages are exchanged Add a new example on how to access performance variables from within the code Allocate arrays regarding value returned by binding Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add overhead benchmark, with script to use data and create graphs out of the results Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix segfault error at end when not loading pml Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Start create common monitoring module. Factorise version numbering Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix microbenchmarks script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Improve readability of code NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add osc monitoring component Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error checking if running out of memory in osc_monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Resolve brutal segfault when double freeing filename Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Moving to ompi/mca/common the proper parts of the monitoring system Using common functions instead of pml specific one. Removing pml ones. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add calls to record monitored data from osc. Use common function to translate ranks. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix test_overhead benchmark script distribution Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix linking library with mca/common Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add passive operations in monitoring_test Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix from rank calculation. Add more detailed error messages Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix osc_monitoring mget_message_count function call Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring common output system Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Consistent output file name (with and without MPI_T). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Always output to a file when flushing at pvar_stop(flush). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update the monitoring documentation. Complete informations from HowTo. Fix a few mistake and typos. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use the world_rank for printf's. Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add security check for unique initialization for osc monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the amout of symbols available outside mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Remove use of __sync_* built-ins. Use opal_atomic_* instead. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Deleting now useless file : moved to common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram ditribution of message sizes Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add coll component for collectives communications monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix log10_2 constant initialization. Fix index calculation for histogram array. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add debug info messages to follow more easily initialization steps. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring. monitoring_filter only indicates if filtering is activated. Fix out of range access in histogram. List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t. Remove useless dead code. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Don't install the test scripts. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix missing procs in hashtable. Cache coll monitoring data. * Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer. * Cache monitoring data relative to collectives operations on creation. * Remove double caching. * Use same proc name definition for hash table when inserting and when retrieving. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use intermediate variable to avoid invalid write while retrieving ranks in hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add missing release of the last element in flush_all. Add release of the hashtable in finalize. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Set world_rank from hashtable only if found Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use predefined symbol from opal system to print int Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add automated check (with MPI_Tools) of monitoring. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix procs list caching in common_monitoring_coll_data_t * Fix monitoring_coll_data type definition. * Use size(COMM_WORLD)-1 to determine max number of digits. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Documentation update. Update and then move the latex and README documentation to a more logical place Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add the use of a machine file for overhead benchmark Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Check for out-of-bound write in histogram Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix common_monitoring_cache object init for MPI_COMM_WORLD Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add RDMA benchmarks to test_overhead Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add technical documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adapt to the new definition of communicators Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update expected output in test/monitoring/monitoring_test.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add dumping histogram in edge case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add consistency in header inclusion. Include ompi/mpi/fortran/mpif-h/bindings.h only if needed. Add sanity check before emptying hashtable. Fix typos in documentation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * misc monitoring fixes * test/monitoring: fix test when weak symbols are not available * monitoring: fix a typo and add a missing file in Makefile.am and have monitoring_common.h and monitoring_common_coll.h included in the distro * test/monitoring: cleanup all tests and make distclean a happy panda * test/monitoring: use gettimeofday() if clock_gettime() is unavailable * monitoring: silence misc warnings (#3) Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp> * Cleanups. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Changing int64_t to size_t. Keep the size_t used accross all monitoring components. Adapt the documentation. Remove useless MPI_Request and MPI_Status from monitoring_test.c. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add parameter for RMA test case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the maximum bound computation for proc list dump. Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add communicator-specific monitored collective data reset Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring scripts to the 'make dist' Also install them in the build and the install directories. Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
2017-06-26 19:21:39 +03:00
#
# $1 = filename
# $2 = osc component names
Topic/monitoring (#3109) Add a monitoring PML, OSC and IO. They track all data exchanges between processes, with capability to include or exclude collective traffic. The monitoring infrastructure is driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows. Documentations and examples have been added, as well as a shared library that can be used with LD_PRELOAD and that allows the monitoring of any application. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * add ability to querry pml monitorinting results with MPI Tools interface using performance variables "pml_monitoring_messages_count" and "pml_monitoring_messages_size" Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Fix a convertion problem and add a comment about the lack of component retain in the new component infrastructure. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Allow the pvar to be written by invoking the associated callback. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Various fixes for the monitoring. Allocate all counting arrays in a single allocation Don't delay the initialization (do it at the first add_proc as we know the number of processes in MPI_COMM_WORLD) Add a choice: with or without MPI_T (default). Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Cleanup for the monitoring module. Fixed few bugs, and reshape the operations to prepare for global or communicator-based monitoring. Start integrating support for MPI_T as well as MCA monitoring. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Adding documentation about how to use pml_monitoring component. Document present the use with and without MPI_T. May not reflect exactly how it works right now, but should reflects how it should work in the end. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c. Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Improve monitoring support (including integration with MPI_T) Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output. Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example Set filename only if using mpi tools Adding missing parameters for fprintf in monitoring_flush (for output in std's cases) Fix expected output/results for example header Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer Base whether to output or not on message count, in order to print something if only empty messages are exchanged Add a new example on how to access performance variables from within the code Allocate arrays regarding value returned by binding Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add overhead benchmark, with script to use data and create graphs out of the results Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix segfault error at end when not loading pml Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Start create common monitoring module. Factorise version numbering Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix microbenchmarks script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Improve readability of code NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add osc monitoring component Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error checking if running out of memory in osc_monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Resolve brutal segfault when double freeing filename Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Moving to ompi/mca/common the proper parts of the monitoring system Using common functions instead of pml specific one. Removing pml ones. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add calls to record monitored data from osc. Use common function to translate ranks. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix test_overhead benchmark script distribution Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix linking library with mca/common Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add passive operations in monitoring_test Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix from rank calculation. Add more detailed error messages Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix osc_monitoring mget_message_count function call Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring common output system Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Consistent output file name (with and without MPI_T). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Always output to a file when flushing at pvar_stop(flush). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update the monitoring documentation. Complete informations from HowTo. Fix a few mistake and typos. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use the world_rank for printf's. Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add security check for unique initialization for osc monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the amout of symbols available outside mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Remove use of __sync_* built-ins. Use opal_atomic_* instead. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Deleting now useless file : moved to common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram ditribution of message sizes Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add coll component for collectives communications monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix log10_2 constant initialization. Fix index calculation for histogram array. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add debug info messages to follow more easily initialization steps. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring. monitoring_filter only indicates if filtering is activated. Fix out of range access in histogram. List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t. Remove useless dead code. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Don't install the test scripts. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix missing procs in hashtable. Cache coll monitoring data. * Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer. * Cache monitoring data relative to collectives operations on creation. * Remove double caching. * Use same proc name definition for hash table when inserting and when retrieving. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use intermediate variable to avoid invalid write while retrieving ranks in hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add missing release of the last element in flush_all. Add release of the hashtable in finalize. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Set world_rank from hashtable only if found Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use predefined symbol from opal system to print int Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add automated check (with MPI_Tools) of monitoring. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix procs list caching in common_monitoring_coll_data_t * Fix monitoring_coll_data type definition. * Use size(COMM_WORLD)-1 to determine max number of digits. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Documentation update. Update and then move the latex and README documentation to a more logical place Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add the use of a machine file for overhead benchmark Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Check for out-of-bound write in histogram Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix common_monitoring_cache object init for MPI_COMM_WORLD Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add RDMA benchmarks to test_overhead Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add technical documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adapt to the new definition of communicators Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update expected output in test/monitoring/monitoring_test.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add dumping histogram in edge case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add consistency in header inclusion. Include ompi/mpi/fortran/mpif-h/bindings.h only if needed. Add sanity check before emptying hashtable. Fix typos in documentation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * misc monitoring fixes * test/monitoring: fix test when weak symbols are not available * monitoring: fix a typo and add a missing file in Makefile.am and have monitoring_common.h and monitoring_common_coll.h included in the distro * test/monitoring: cleanup all tests and make distclean a happy panda * test/monitoring: use gettimeofday() if clock_gettime() is unavailable * monitoring: silence misc warnings (#3) Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp> * Cleanups. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Changing int64_t to size_t. Keep the size_t used accross all monitoring components. Adapt the documentation. Remove useless MPI_Request and MPI_Status from monitoring_test.c. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add parameter for RMA test case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the maximum bound computation for proc list dump. Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add communicator-specific monitored collective data reset Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring scripts to the 'make dist' Also install them in the build and the install directories. Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
2017-06-26 19:21:39 +03:00
#
AC_DEFUN(
[MCA_OMPI_OSC_MONITORING_GENERATE_TEMPLATES],
[m4_ifval(
[$1],
[AC_CONFIG_COMMANDS(
[$1],
[filename="$1"
components=`echo "$2" | sed -e 's/,/ /g' -e 's/monitoring//'`
cat <<EOF >$filename
/* $filename
*
* This file was generated from ompi/mca/osc/monitoring/configure.m4
*
* DO NOT EDIT THIS FILE.
*
*/
/*
* Copyright (c) 2017-2018 Inria. All rights reserved.
* \$COPYRIGHT$
*
* Additional copyrights may follow
*
* \$HEADER$
*/
#ifndef MCA_OSC_MONITORING_GEN_TEMPLATE_H
#define MCA_OSC_MONITORING_GEN_TEMPLATE_H
#include <ompi_config.h>
#include <ompi/mca/osc/osc.h>
#include <ompi/mca/osc/monitoring/osc_monitoring_template.h>
/************************************************************/
/* Include template generating macros and inlined functions */
EOF
# Generate each case in order to register the proper template functions
for comp in $components
do
echo "OSC_MONITORING_MODULE_TEMPLATE_GENERATE(${comp})" >>$filename
done
cat <<EOF >>$filename
/************************************************************/
typedef struct {
const char * name;
ompi_osc_base_module_t * (*fct) (ompi_osc_base_module_t *);
} osc_monitoring_components_list_t;
static const osc_monitoring_components_list_t osc_monitoring_components_list[[]] = {
EOF
for comp in $components
do
echo " { .name = \"${comp}\", .fct = OSC_MONITORING_SET_TEMPLATE_FCT_NAME(${comp}) }," >>$filename
done
cat <<EOF >>$filename
{ .name = NULL, .fct = NULL }
};
#endif /* MCA_OSC_MONITORING_GEN_TEMPLATE_H */
EOF
unset filename components
])
])dnl
])dnl
Topic/monitoring (#3109) Add a monitoring PML, OSC and IO. They track all data exchanges between processes, with capability to include or exclude collective traffic. The monitoring infrastructure is driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows. Documentations and examples have been added, as well as a shared library that can be used with LD_PRELOAD and that allows the monitoring of any application. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * add ability to querry pml monitorinting results with MPI Tools interface using performance variables "pml_monitoring_messages_count" and "pml_monitoring_messages_size" Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Fix a convertion problem and add a comment about the lack of component retain in the new component infrastructure. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Allow the pvar to be written by invoking the associated callback. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Various fixes for the monitoring. Allocate all counting arrays in a single allocation Don't delay the initialization (do it at the first add_proc as we know the number of processes in MPI_COMM_WORLD) Add a choice: with or without MPI_T (default). Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Cleanup for the monitoring module. Fixed few bugs, and reshape the operations to prepare for global or communicator-based monitoring. Start integrating support for MPI_T as well as MCA monitoring. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Adding documentation about how to use pml_monitoring component. Document present the use with and without MPI_T. May not reflect exactly how it works right now, but should reflects how it should work in the end. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c. Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Improve monitoring support (including integration with MPI_T) Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output. Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example Set filename only if using mpi tools Adding missing parameters for fprintf in monitoring_flush (for output in std's cases) Fix expected output/results for example header Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer Base whether to output or not on message count, in order to print something if only empty messages are exchanged Add a new example on how to access performance variables from within the code Allocate arrays regarding value returned by binding Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add overhead benchmark, with script to use data and create graphs out of the results Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix segfault error at end when not loading pml Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Start create common monitoring module. Factorise version numbering Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix microbenchmarks script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Improve readability of code NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add osc monitoring component Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error checking if running out of memory in osc_monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Resolve brutal segfault when double freeing filename Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Moving to ompi/mca/common the proper parts of the monitoring system Using common functions instead of pml specific one. Removing pml ones. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add calls to record monitored data from osc. Use common function to translate ranks. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix test_overhead benchmark script distribution Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix linking library with mca/common Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add passive operations in monitoring_test Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix from rank calculation. Add more detailed error messages Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix osc_monitoring mget_message_count function call Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring common output system Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Consistent output file name (with and without MPI_T). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Always output to a file when flushing at pvar_stop(flush). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update the monitoring documentation. Complete informations from HowTo. Fix a few mistake and typos. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use the world_rank for printf's. Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add security check for unique initialization for osc monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the amout of symbols available outside mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Remove use of __sync_* built-ins. Use opal_atomic_* instead. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Deleting now useless file : moved to common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram ditribution of message sizes Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add coll component for collectives communications monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix log10_2 constant initialization. Fix index calculation for histogram array. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add debug info messages to follow more easily initialization steps. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring. monitoring_filter only indicates if filtering is activated. Fix out of range access in histogram. List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t. Remove useless dead code. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Don't install the test scripts. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix missing procs in hashtable. Cache coll monitoring data. * Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer. * Cache monitoring data relative to collectives operations on creation. * Remove double caching. * Use same proc name definition for hash table when inserting and when retrieving. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use intermediate variable to avoid invalid write while retrieving ranks in hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add missing release of the last element in flush_all. Add release of the hashtable in finalize. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Set world_rank from hashtable only if found Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Use predefined symbol from opal system to print int Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add automated check (with MPI_Tools) of monitoring. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix procs list caching in common_monitoring_coll_data_t * Fix monitoring_coll_data type definition. * Use size(COMM_WORLD)-1 to determine max number of digits. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Documentation update. Update and then move the latex and README documentation to a more logical place Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add the use of a machine file for overhead benchmark Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Check for out-of-bound write in histogram Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Fix common_monitoring_cache object init for MPI_COMM_WORLD Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add RDMA benchmarks to test_overhead Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2). Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add technical documentation Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adapt to the new definition of communicators Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Update expected output in test/monitoring/monitoring_test.c Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add dumping histogram in edge case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add consistency in header inclusion. Include ompi/mpi/fortran/mpif-h/bindings.h only if needed. Add sanity check before emptying hashtable. Fix typos in documentation. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * misc monitoring fixes * test/monitoring: fix test when weak symbols are not available * monitoring: fix a typo and add a missing file in Makefile.am and have monitoring_common.h and monitoring_common_coll.h included in the distro * test/monitoring: cleanup all tests and make distclean a happy panda * test/monitoring: use gettimeofday() if clock_gettime() is unavailable * monitoring: silence misc warnings (#3) Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp> * Cleanups. Signed-off-by: George Bosilca <bosilca@icl.utk.edu> * Changing int64_t to size_t. Keep the size_t used accross all monitoring components. Adapt the documentation. Remove useless MPI_Request and MPI_Status from monitoring_test.c. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add parameter for RMA test case Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Clean the maximum bound computation for proc list dump. Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6. Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add communicator-specific monitored collective data reset Signed-off-by: Clement Foyer <clement.foyer@inria.fr> * Add monitoring scripts to the 'make dist' Also install them in the build and the install directories. Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
2017-06-26 19:21:39 +03:00
# MCA_ompi_osc_monitoring_CONFIG()
# ------------------------------------------------
AC_DEFUN(
[MCA_ompi_osc_monitoring_CONFIG],
[AC_CONFIG_FILES([ompi/mca/osc/monitoring/Makefile])
AS_IF([test "$MCA_BUILD_ompi_common_monitoring_DSO_TRUE" = ''],
[$1],
[$2])
MCA_OMPI_OSC_MONITORING_GENERATE_TEMPLATES(
[ompi/mca/osc/monitoring/osc_monitoring_template_gen.h],
[mca_ompi_osc_m4_config_component_list, mca_ompi_osc_no_config_component_list])dnl
])dnl