call the opal_common_verbs_mca_register function to make sure that
opal_common_verbs_want_fork_support mca parameter is created and therefore
can be used to control the fork support.
Both opal_shmem_base_select() and
opal_shmem_base_best_runnable_component_name() and were calling
opal_shmem_base_runtime_query(), which would do component selection
(and closing of losing components) twice.
Put protection in opal_shmem_base_runtime_query() to return the cached
results the second time. Additionally, make
opal_shmem_base_runtime_query() "own" the cached results
(vs. opal_shmem_base_select).
This allows the opal_shmem_base_module_t to be properly cast to an
mca_base_module_t.
(this commit is the rationale for the previous shmem C99 .member
initialization commit)
Also include two other minor changes:
1. More C99-style member initialization in the component struct
1. Fix the BTL module member initialization to not be redundant
Due to the nature of the cache architecture on power,
we don't export coherency_line_size for L2 in sysfs.
If we are unable to get the L2 cache line size, try L1.
See open-mpi/ompi#383 for more information.
In order to have an effect, ibv_fork_init should be called in the
beginning of the verbs initialization flow - before the calls to the
ibv_create_qp and ibv_create_cq verbs.
These functions are called from the oob/ud code and by the time the
other verbs components (btl openib, pml yalla, ...) call ibv_fork_init,
it's too late. This commit forces the call to ibv_fork_init (if it's
requested) right at the beginning of all the components that are using
verbs.
(ibv_fork_init() can be safely called multiple times)
This commit also removes the btl_openib_want_fork_support mca parameter
and adds a new mca parameter instead - opal_verbs_want_fork_support.
Through this new parameter, fork support may be requested for ALL
components.
The default value for this parameter is set to 1.
Before this commit the btl_openib_want_fork_support parameter didn't
provide fork support for the openib btl if its value was set to 1.
(because when openib called ibv_fork_init, it was already after the
calls to ibv_create_* in oob/ud and thereofre it failed).
Use of the old ompi_free_list_t and ompi_free_list_item_t is
deprecated. These classes will be removed in a future commit.
This commit updates the entire code base to use opal_free_list_t and
opal_free_list_item_t.
Notes:
OMPI_FREE_LIST_*_MT -> opal_free_list_* (uses opal_using_threads ())
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Please verify your components have been updated correctly. Keep in
mind that in terms of threading:
OPAL_FREE_LIST_GET -> opal_free_list_get_st
OPAL_FREE_LIST_RETURN -> opal_free_list_return_st
I used the opal_using_threads() variant anytime it appeared multiple
threads could be operating on the free list. If this is not the case
update to _st. If multiple threads are always in use change to _mt.
Historically these two lists were different due to ompi_free_list_t
dependencies in ompi (mpool). Those dependencies have since been moved
to opal so it is safe to (finally) combine them. The combined free
list comes in three flavors:
- Single-threaded. Only to be used when it is guaranteed that no
concurrent access will be made to the free list. Single-threaded
functions are suffixed with _st.
- Mutli-threaded. To be used when the free list may be accessed by
multiple threads despite the setting of opal_using_threads.
Multi-threaded functins are suffixed with _mt.
- Conditionally multi-threaded. Common use case. These functions are
thread-safe if opal_using_threads is set to true.
Compatibility functions for the ompi_free_list_t and the old accessor
functions (OPAL_FREE_LIST_*) are available while the code base is
transitioned to the new class/functions.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
