Passed the below set of symbols into a script that added ompi_ to them all.
Note that if processing a symbol named "foo" the script turns
foo into ompi_foo
but doesn't turn
foobar into ompi_foobar
But beyond that the script is blind to C syntax, so it hits strings and
comments etc as well as vars/functions.
coll_base_comm_get_reqs
comm_allgather_pml
comm_allreduce_pml
comm_bcast_pml
fcoll_base_coll_allgather_array
fcoll_base_coll_allgatherv_array
fcoll_base_coll_bcast_array
fcoll_base_coll_gather_array
fcoll_base_coll_gatherv_array
fcoll_base_coll_scatterv_array
fcoll_base_sort_iovec
mpit_big_lock
mpit_init_count
mpit_lock
mpit_unlock
netpatterns_base_err
netpatterns_base_verbose
netpatterns_cleanup_narray_knomial_tree
netpatterns_cleanup_recursive_doubling_tree_node
netpatterns_cleanup_recursive_knomial_allgather_tree_node
netpatterns_cleanup_recursive_knomial_tree_node
netpatterns_init
netpatterns_register_mca_params
netpatterns_setup_multinomial_tree
netpatterns_setup_narray_knomial_tree
netpatterns_setup_narray_tree
netpatterns_setup_narray_tree_contigous_ranks
netpatterns_setup_recursive_doubling_n_tree_node
netpatterns_setup_recursive_doubling_tree_node
netpatterns_setup_recursive_knomial_allgather_tree_node
netpatterns_setup_recursive_knomial_tree_node
pml_v_output_close
pml_v_output_open
intercept_extra_state_t
odls_base_default_wait_local_proc
_event_debug_mode_on
_evthread_cond_fns
_evthread_id_fn
_evthread_lock_debugging_enabled
_evthread_lock_fns
cmd_line_option_t
cmd_line_param_t
crs_base_self_checkpoint_fn
crs_base_self_continue_fn
crs_base_self_restart_fn
event_enable_debug_output
event_global_current_base_
event_module_include
eventops
sync_wait_mt
trigger_user_inc_callback
var_type_names
var_type_sizes
Signed-off-by: Mark Allen <markalle@us.ibm.com>
Add a monitoring PML, OSC and IO. They track all data exchanges between processes,
with capability to include or exclude collective traffic. The monitoring infrastructure is
driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows.
Documentations and examples have been added, as well as a shared library that can be
used with LD_PRELOAD and that allows the monitoring of any application.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* add ability to querry pml monitorinting results with MPI Tools interface
using performance variables "pml_monitoring_messages_count" and
"pml_monitoring_messages_size"
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Fix a convertion problem and add a comment about the lack of component
retain in the new component infrastructure.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Allow the pvar to be written by invoking the associated callback.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Various fixes for the monitoring.
Allocate all counting arrays in a single allocation
Don't delay the initialization (do it at the first add_proc as we
know the number of processes in MPI_COMM_WORLD)
Add a choice: with or without MPI_T (default).
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Cleanup for the monitoring module.
Fixed few bugs, and reshape the operations to prepare for
global or communicator-based monitoring. Start integrating
support for MPI_T as well as MCA monitoring.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Adding documentation about how to use pml_monitoring component.
Document present the use with and without MPI_T.
May not reflect exactly how it works right now, but should reflects
how it should work in the end.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c.
Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Improve monitoring support (including integration with MPI_T)
Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set
Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename
Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output.
Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example
Set filename only if using mpi tools
Adding missing parameters for fprintf in monitoring_flush (for output in std's cases)
Fix expected output/results for example header
Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer
Base whether to output or not on message count, in order to print something if only empty messages are exchanged
Add a new example on how to access performance variables from within the code
Allocate arrays regarding value returned by binding
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add overhead benchmark, with script to use data and create graphs out of the results
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix segfault error at end when not loading pml
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Start create common monitoring module. Factorise version numbering
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix microbenchmarks script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Improve readability of code
NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add osc monitoring component
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add error checking if running out of memory in osc_monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Resolve brutal segfault when double freeing filename
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Moving to ompi/mca/common the proper parts of the monitoring system
Using common functions instead of pml specific one. Removing pml ones.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add calls to record monitored data from osc. Use common function to translate ranks.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix test_overhead benchmark script distribution
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix linking library with mca/common
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add passive operations in monitoring_test
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix from rank calculation. Add more detailed error messages
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix osc_monitoring mget_message_count function call
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add monitoring common output system
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Consistent output file name (with and without MPI_T).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Always output to a file when flushing at pvar_stop(flush).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Update the monitoring documentation.
Complete informations from HowTo. Fix a few mistake and typos.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use the world_rank for printf's.
Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add security check for unique initialization for osc monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean the amout of symbols available outside mca/common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Remove use of __sync_* built-ins. Use opal_atomic_* instead.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Deleting now useless file : moved to common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add histogram ditribution of message sizes
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add coll component for collectives communications monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix log10_2 constant initialization. Fix index calculation for histogram array.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add debug info messages to follow more easily initialization steps.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring.
monitoring_filter only indicates if filtering is activated.
Fix out of range access in histogram.
List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t.
Remove useless dead code.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Don't install the test scripts.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix missing procs in hashtable. Cache coll monitoring data.
* Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer.
* Cache monitoring data relative to collectives operations on creation.
* Remove double caching.
* Use same proc name definition for hash table when inserting and
when retrieving.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use intermediate variable to avoid invalid write while retrieving ranks in hashtable.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add missing release of the last element in flush_all. Add release of the hashtable in finalize.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Set world_rank from hashtable only if found
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use predefined symbol from opal system to print int
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add automated check (with MPI_Tools) of monitoring.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix procs list caching in common_monitoring_coll_data_t
* Fix monitoring_coll_data type definition.
* Use size(COMM_WORLD)-1 to determine max number of digits.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Documentation update.
Update and then move the latex and README documentation to a more logical place
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add the use of a machine file for overhead benchmark
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Check for out-of-bound write in histogram
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix common_monitoring_cache object init for MPI_COMM_WORLD
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add RDMA benchmarks to test_overhead
Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add technical documentation
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Adapt to the new definition of communicators
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Update expected output in test/monitoring/monitoring_test.c
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add dumping histogram in edge case
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add consistency in header inclusion.
Include ompi/mpi/fortran/mpif-h/bindings.h only if needed.
Add sanity check before emptying hashtable.
Fix typos in documentation.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* misc monitoring fixes
* test/monitoring: fix test when weak symbols are not available
* monitoring: fix a typo and add a missing file in Makefile.am
and have monitoring_common.h and monitoring_common_coll.h included in the distro
* test/monitoring: cleanup all tests and make distclean a happy panda
* test/monitoring: use gettimeofday() if clock_gettime() is unavailable
* monitoring: silence misc warnings (#3)
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
* Cleanups.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Changing int64_t to size_t.
Keep the size_t used accross all monitoring components.
Adapt the documentation.
Remove useless MPI_Request and MPI_Status from monitoring_test.c.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add parameter for RMA test case
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean the maximum bound computation for proc list dump.
Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add communicator-specific monitored collective data reset
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add monitoring scripts to the 'make dist'
Also install them in the build and the install directories.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
since Open MPI now requires a C99, and ptrdiff_t type is part of C99,
there is no more need for the abstract OPAL_PTRDIFF_TYPE type.
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
- Support MPI-2.2 and MPI-3.0 COLL features.
* `MPI_REDUCE_SCATTER_BLOCK`
* neighborhood collective communication
* nonblocking collective communication
- Add `*_BASE_ARGS` and `*_BASE_ARG_NAMES` for convenience.
- Use parameter names used in the MPI Standard.
Signed-off-by: KAWASHIMA Takahiro <t-kawashima@jp.fujitsu.com>
As we changed the ABI (forcing a major release), we can limit
the size of the predefined communicators by moving the collective
structure outside the communicator. This might have a minimal,
but unnoticeable, impact on performance. This approach has been
discussed during the January 2017 devel meeting.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
* Since we are adding a new function to `mca_coll_base_module_2_1_0_t`
we need to increase the version of the module structure to `2_2_0`.
* Add a comment just above the PREDEFINED_COMMUNICATOR_PAD describing
it's purpose and when it should change. To help future developers
trying to answer the question noted in the comment.
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
MPI_Allgatherv with MPI_IN_PLACE reads data from wrong location.
They were locating the MPI_IN_PLACE send buffer as
```c
send_buf = (char*)rbuf;
for (i = 0; i < rank; ++i) {
send_buf += ((ptrdiff_t)rcounts[i] * extent);
}
```
when it should be
```c
send_buf = (char*)rbuf;
send_buf += ((ptrdiff_t)disps[rank] * extent);
```
because disps[] specifies where things are in the v-style buffers.
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
- instead of coll_base_comm_get_reqs(2) for irecv/isend, use only
one request allocated in the stack and do a irecv/send
- instead of ompi_request_wait_all(2), simpy ompi_request_wait
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
Clang 5.1 on my mac was a sad panda compiling a couple
of files, complaining about uninitialized stack variables.
This commit makes clang a happier panda (or at least not so sad).
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Based on current implementation it is faster to use a blocking
send than the non-blocking version. Switch the exchange function
used in the barrier to use the blocking version combined with
the non-blocking version of the receive.
This is similar to open-mpi/ompi@223d75595d
* Print a verbose message if the component was disqualified because of
a negative priority.
* If a disqualified component provided a module, release it.
* Display list of selected components in priority order
- During the process of volunteering collective functions for a
communicator, print the component name and priority. This will
cause the verbose messages to be displayed in reverse priority
order (lowest priority first, up to highest). This is helpful
when determining which collective components are active in which
order for a given communicator.
To see the messages you need the following MCA parameter set to 9
or higher: `-mca coll_base_verbose 9`
* Adjust verbose for commonly needed verbose output from 10 to 9 to
make it easier to access this information.
Based on current implementation it is faster to use a blocking
send than the non-blocking version. Switch the exchange function
used in the barrier to use the blocking version combined with
the non-blocking version of the receive.
Or at least that was the origin of the issue. It turns out
we were freeing the wrong buffer (but as it only happen in the
case of an error we never noticed).
This patch addresses most (if not all) @derbeyn concerns
expressed on #1015. I added checks for the requests allocation
in all functions, ompi_coll_base_free_reqs is called with the
right number of requests, I removed the unnecessary basic_module_comm_t
and use the base_module_comm_t instead, I remove all uses of the
COLL_BASE_BCAST_USE_BLOCKING define, and other minor fixes.
Fix CID 1325868 (#1 of 1): Dereference after null check (FORWARD_NULL):
Fix CID 1325869 (#1-2 of 2): Dereference after null check (FORWARD_NULL):
Here reqs can indeed be NULL. Added a check to
ompi_coll_base_free_reqs to prevent dereferencing NULL pointer.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
some of the collective modules. Added a new function
opan_datatype_span, to compute the memory span of
count number of datatype, excluding the gaps in the
beginning and at the end. If a memory allocation is
made using the returned value, the gap (also returned)
should be removed from the allocated pointer.
