This commit enables MCA support for the following PSM2 environment
variables: PSM2_DEVICES, PSM2_MEMORY, PSM2_MQ_SENDREQS_MAX,
PSM2_MQ_RECVREQS_MAX, PSM2_MQ_RNDV_HFI_THRESH,
PSM2_MQ_RNDV_SHM_THRESH, PSM2_RCVTHREAD, PSM2_SHAREDCONTEXTS,
PSM2_SHAREDCONTEXTS_MAX, and PSM2_TRACEMASK. These variable can be set
by MCA if they are not already set in the environment.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds a helper function to get the inbound and outbound
neighbor count and updates the neighbor_allgatherv bindings to use the
correct count when checking the input parameters.
Fixes#2324
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
There is no reason not to progress OSC during the MPI_Win_flush_local
and MPI_Win_flush_all_local calls. This fixes#3750.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit fixes a bug that occurs when the btl callback happens before
the rget returns. In this case the fragment has been returned and is no
longer valid. This commit saves the size before calling rget. This is
valid since the BTL is not allowed to change the read size.
Fixes#3821
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Passed the below set of symbols into a script that added ompi_ to them all.
Note that if processing a symbol named "foo" the script turns
foo into ompi_foo
but doesn't turn
foobar into ompi_foobar
But beyond that the script is blind to C syntax, so it hits strings and
comments etc as well as vars/functions.
coll_base_comm_get_reqs
comm_allgather_pml
comm_allreduce_pml
comm_bcast_pml
fcoll_base_coll_allgather_array
fcoll_base_coll_allgatherv_array
fcoll_base_coll_bcast_array
fcoll_base_coll_gather_array
fcoll_base_coll_gatherv_array
fcoll_base_coll_scatterv_array
fcoll_base_sort_iovec
mpit_big_lock
mpit_init_count
mpit_lock
mpit_unlock
netpatterns_base_err
netpatterns_base_verbose
netpatterns_cleanup_narray_knomial_tree
netpatterns_cleanup_recursive_doubling_tree_node
netpatterns_cleanup_recursive_knomial_allgather_tree_node
netpatterns_cleanup_recursive_knomial_tree_node
netpatterns_init
netpatterns_register_mca_params
netpatterns_setup_multinomial_tree
netpatterns_setup_narray_knomial_tree
netpatterns_setup_narray_tree
netpatterns_setup_narray_tree_contigous_ranks
netpatterns_setup_recursive_doubling_n_tree_node
netpatterns_setup_recursive_doubling_tree_node
netpatterns_setup_recursive_knomial_allgather_tree_node
netpatterns_setup_recursive_knomial_tree_node
pml_v_output_close
pml_v_output_open
intercept_extra_state_t
odls_base_default_wait_local_proc
_event_debug_mode_on
_evthread_cond_fns
_evthread_id_fn
_evthread_lock_debugging_enabled
_evthread_lock_fns
cmd_line_option_t
cmd_line_param_t
crs_base_self_checkpoint_fn
crs_base_self_continue_fn
crs_base_self_restart_fn
event_enable_debug_output
event_global_current_base_
event_module_include
eventops
sync_wait_mt
trigger_user_inc_callback
var_type_names
var_type_sizes
Signed-off-by: Mark Allen <markalle@us.ibm.com>
Rearrange the receive frag timeout logic to avoid calling
opal_timer_base_get_usec() in read_msg(). Instead set it at the first
retry.
Signed-off-by: Todd Kordenbrock <thkgcode@gmail.com>
If the a frag cannot be retried because the ni_fail_type is other than
PTL_NI_DROPPED, then set the return type and jump to callback_error.
This sets MPI_ERROR and completes the receive.
Signed-off-by: Todd Kordenbrock <thkgcode@gmail.com>
This commit changes the locking code to allow the lock release to be
non-blocking. This helps with releasing the accumulate lock which may
occur in a BTL callback.
Fixes#3616
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Add a monitoring PML, OSC and IO. They track all data exchanges between processes,
with capability to include or exclude collective traffic. The monitoring infrastructure is
driven using MPI_T, and can be tuned of and on any time o any communicators/files/windows.
Documentations and examples have been added, as well as a shared library that can be
used with LD_PRELOAD and that allows the monitoring of any application.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* add ability to querry pml monitorinting results with MPI Tools interface
using performance variables "pml_monitoring_messages_count" and
"pml_monitoring_messages_size"
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Fix a convertion problem and add a comment about the lack of component
retain in the new component infrastructure.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Allow the pvar to be written by invoking the associated callback.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Various fixes for the monitoring.
Allocate all counting arrays in a single allocation
Don't delay the initialization (do it at the first add_proc as we
know the number of processes in MPI_COMM_WORLD)
Add a choice: with or without MPI_T (default).
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Cleanup for the monitoring module.
Fixed few bugs, and reshape the operations to prepare for
global or communicator-based monitoring. Start integrating
support for MPI_T as well as MCA monitoring.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Adding documentation about how to use pml_monitoring component.
Document present the use with and without MPI_T.
May not reflect exactly how it works right now, but should reflects
how it should work in the end.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Change rank into MPI_COMM_WORLD and size(MPI_COMM_WORLD) to global variables in pml_monitoring.c.
Change mca_pml_monitoring_flush() signature so we don't need the size and rank parameters.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Improve monitoring support (including integration with MPI_T)
Use mca_pml_monitoring_enable to check status state. Set mca_pml_monitoring_current_filename iif parameter is set
Allow 3 modes for pml_monitoring_enable_output: - 1 : stdout; - 2 : stderr; - 3 : filename
Fix test : 1 for differenciated messages, >1 for not differenciated. Fix output.
Add documentation for pml_monitoring_enable_output parameter. Remove useless parameter in example
Set filename only if using mpi tools
Adding missing parameters for fprintf in monitoring_flush (for output in std's cases)
Fix expected output/results for example header
Fix exemple when using MPI_Tools : a null-pointer can't be passed directly. It needs to be a pointer to a null-pointer
Base whether to output or not on message count, in order to print something if only empty messages are exchanged
Add a new example on how to access performance variables from within the code
Allocate arrays regarding value returned by binding
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add overhead benchmark, with script to use data and create graphs out of the results
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix segfault error at end when not loading pml
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Start create common monitoring module. Factorise version numbering
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix microbenchmarks script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Improve readability of code
NULL can't be passed as a PVAR parameter value. It must be a pointer to NULL or an empty string.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add osc monitoring component
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add error checking if running out of memory in osc_monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Resolve brutal segfault when double freeing filename
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Moving to ompi/mca/common the proper parts of the monitoring system
Using common functions instead of pml specific one. Removing pml ones.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add calls to record monitored data from osc. Use common function to translate ranks.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix test_overhead benchmark script distribution
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix linking library with mca/common
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add passive operations in monitoring_test
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix from rank calculation. Add more detailed error messages
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix alignments. Fix common_monitoring_get_world_rank function. Remove useless trailing new lines
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix osc_monitoring mget_message_count function call
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Change common_monitoring function names to respect the naming convention. Move to common_finalize the common parts of finalization. Add some comments.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add monitoring common output system
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add error message when trying to flush to a file, and open fails. Remove erroneous info message when flushing wereas the monitoring is already disabled.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Consistent output file name (with and without MPI_T).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Always output to a file when flushing at pvar_stop(flush).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Update the monitoring documentation.
Complete informations from HowTo. Fix a few mistake and typos.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use the world_rank for printf's.
Fix name generation for output files when using MPI_T. Minor changes in benchmarks starting script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean potential previous runs, but keep the results at the end in order to potentially reprocess the data. Add comments.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add security check for unique initialization for osc monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean the amout of symbols available outside mca/common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Remove use of __sync_* built-ins. Use opal_atomic_* instead.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Allocate the hashtable on common/monitoring component initialization. Define symbols to set the values for error/warning/info verbose output. Use opal_atomic instead of built-in function in osc/monitoring template initialization.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Deleting now useless file : moved to common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add histogram ditribution of message sizes
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add histogram array of 2-based log of message sizes. Use simple call to reset/allocate arrays in common_monitoring.c
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add informations in dumping file. Separate per category (pt2pt/osc/coll (to come)) monitored data
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add coll component for collectives communications monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix warning messages : use c_name as the magic id is not always defined. Moreover, there was a % missing. Add call to release underlying modules. Add debug info messages. Add warning which may lead to further analysis.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix log10_2 constant initialization. Fix index calculation for histogram array.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add debug info messages to follow more easily initialization steps.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Group all the var/pvar definitions to common_monitoring. Separate initial filename from the current on, to ease its lifetime management. Add verifications to ensure common is initialized once only. Move state variable management to common_monitoring.
monitoring_filter only indicates if filtering is activated.
Fix out of range access in histogram.
List is not used with the struct mca_monitoring_coll_data_t, so heritate only from opal_object_t.
Remove useless dead code.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix invalid memory allocation. Initialize initial_filename to empty string to avoid invalid read in mca_base_var_register.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Don't install the test scripts.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix missing procs in hashtable. Cache coll monitoring data.
* Add MCA_PML_BASE_FLAG_REQUIRE_WORLD flag to the PML layer.
* Cache monitoring data relative to collectives operations on creation.
* Remove double caching.
* Use same proc name definition for hash table when inserting and
when retrieving.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use intermediate variable to avoid invalid write while retrieving ranks in hashtable.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add missing release of the last element in flush_all. Add release of the hashtable in finalize.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use a linked list instead of a hashtable to keep tracks of communicator data. Add release of the structure at finalize time.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Set world_rank from hashtable only if found
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Use predefined symbol from opal system to print int
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Move collective monitoring data to a hashtable. Add pvar to access the monitoring_coll_data. Move functions header to a private file only to be used in ompi/mca/common/monitoring
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix pvar registration. Use OMPI_ERROR isntead of -1 as returned error value. Fix releasing of coll_data_t objects. Affect value only if data is found in the hashtable.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add automated check (with MPI_Tools) of monitoring.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix procs list caching in common_monitoring_coll_data_t
* Fix monitoring_coll_data type definition.
* Use size(COMM_WORLD)-1 to determine max number of digits.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add linking to Fortran applications for LD_PRELOAD usage of monitoring_prof
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add PVAR's handles. Clean up code (visibility, add comments...). Start updating the documentation
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix coll operations monitoring. Update check_monitoring accordingly to the added pvar. Fix monitoring array allocation.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Documentation update.
Update and then move the latex and README documentation to a more logical place
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Aggregate monitoring COLL data to the generated matrix. Update documentation accordingly.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix monitoring_prof (bad variable.vector used, and wrong array in PMPI_Gather).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add reduce_scatter and reduce_scatter_block monitoring. Reduce memory footprint of monitoring_prof. Unify OSC related outputs.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add the use of a machine file for overhead benchmark
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Check for out-of-bound write in histogram
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Fix common_monitoring_cache object init for MPI_COMM_WORLD
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add RDMA benchmarks to test_overhead
Add error file output. Add MPI_Put and MPI_Get results analysis. Add overhead computation for complete sending (pingpong / 2).
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add computation of average and median of overheads. Add comments and copyrigths to the test_overhead script
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add technical documentation
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Adapt to the new definition of communicators
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Update expected output in test/monitoring/monitoring_test.c
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add dumping histogram in edge case
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Adding a reduce(pml_monitoring_messages_count, MPI_MAX) example
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add consistency in header inclusion.
Include ompi/mpi/fortran/mpif-h/bindings.h only if needed.
Add sanity check before emptying hashtable.
Fix typos in documentation.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* misc monitoring fixes
* test/monitoring: fix test when weak symbols are not available
* monitoring: fix a typo and add a missing file in Makefile.am
and have monitoring_common.h and monitoring_common_coll.h included in the distro
* test/monitoring: cleanup all tests and make distclean a happy panda
* test/monitoring: use gettimeofday() if clock_gettime() is unavailable
* monitoring: silence misc warnings (#3)
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
* Cleanups.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
* Changing int64_t to size_t.
Keep the size_t used accross all monitoring components.
Adapt the documentation.
Remove useless MPI_Request and MPI_Status from monitoring_test.c.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add parameter for RMA test case
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Clean the maximum bound computation for proc list dump.
Use ptrdiff_t instead of OPAL_PTRDIFF_TYPE to reflect the changes from commit fa5cd0dbe5d261bd9d2cc61d5b305b4ef6a2dda6.
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add communicator-specific monitored collective data reset
Signed-off-by: Clement Foyer <clement.foyer@inria.fr>
* Add monitoring scripts to the 'make dist'
Also install them in the build and the install directories.
Signed-off-by: George Bosilca <bosilca@icl.utk.edu>
The data endpoint was not being set correctly for local peers in some
cases. This commit fixes the bug and cleans the associated code to
simplify the logic.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
make NBC_Handle (almost) an internal structure created
by NBC_Schedule_request()
use a local variable instead of what was previously handle->tmpbuf
Refs open-mpi/ompi#3487
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
update the cartesian communicator based grouping strategy to match the other
algorithms used in the aggregator selection process.
Signed-off-by: Edgar Gabriel <egabriel@central.uh.edu>
the cart_based_grouping aggregator strategy was not correctly updated
during the last major rewrite of the aggregator selection algorithm.
It is also not supposed to be called from file_open (but from
file_set_view).
Signed-off-by: Edgar Gabriel <egabriel@central.uh.edu>
* Add work around support for the following missing ops in ROMIO 3.1.4
- `MPI_File_iread_at_all`
- `MPI_File_iwrite_at_all`
- `MPI_File_iread_all`
- `MPI_File_iwrite_all`
Signed-off-by: Joshua Hursey <jhursey@us.ibm.com>
This commit adds code to allow support for the info assertions added
by mpi-forum/mpi-issues#11. The assertions added are:
mpi_assert_no_any_tag, mpi_assert_no_any_source,
mpi_assert_exact_length, and mpi_assert_allow_overtaking.
This commit also adds support for the mpi_assert_no_any_source and
mpi_assert_allow_overtaking info keys to the ob1 pml.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
This commit adds the `req_start` member to the `ompi_request_t` struct.
The `MPI_START` and `MPI_STARTALL` routines call this callback function
instead of `MCA_PML_CALL(start(...))`. So components that return
persistent request must set this member to their request objects.
`mca_pml_base_module_t::pml_start` is not deleted because
`MCA_PML_CALL(start(...))` is still used elsewhere across OMPI.
Signed-off-by: KAWASHIMA Takahiro <t-kawashima@jp.fujitsu.com>
This commit moves the info subscribe for the blocking_fence to after
the global_state is allocated and moves setting win->w_osc_module to
before the info subscribe for alloc_shared_contig. This fixes a SEGV
caught by MTT.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
a buffer defined by (buf, count, dt)
will have data starting at buf+offset and ending len bytes later with
len = opal_datatype_span(&dt.super, count, &offset);
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
A component with implementation of R. Rabenseifner's algorithm for Reduce and Allreduce.
This algorithm is a combination of a reduce-scatter implemented with recursive vector halving
and recursive distance doubling, followed either by a gather or an allgather.
Current limitations:
-- count >= 2^{\floor{\log_2 p}}
-- commutative operations only
-- intra-communicators onl
Signed-off-by: Mikhail Kurnosov <mkurnosov@gmail.com>
coll/spacc: Modify implementation to use `ompi_coll_base_sendrecv()`
Replace irecv() + isend() + ompi_request_wait() to ompi_coll_base_sendrecv().
Signed-off-by: Mikhail Kurnosov <mkurnosov@gmail.com>
origin_datatype and target_datatype might be different and hence have different extent,
so use either origin_extent or target_extent when appropriate.
Refs open-mpi/ompi#3569
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
