1. minor modification to include two new opal MCA params:
(a) opal_profile: outputs what components were selected by each framework
currently enabled for most, but not all, frameworks
(b) opal_profile_file: name of file that contains profile info required
for modex
2. introduction of two new tools:
(a) ompi-probe: MPI process that simply calls MPI_Init/Finalize with
opal_profile set. Also reports back the rml IP address for all
interfaces on the node
(b) ompi-profiler: uses ompi-probe to create the profile_file, also
reports out a summary of what framework components are actually
being used to help with configuration options
3. modification of the grpcomm basic component to utilize the
profile file in place of the modex where possible
4. modification of orterun so it properly sees opal mca params and
handles opal_profile correctly to ensure we don't get its profile
5. similar mod to orted as for orterun
6. addition of new test that calls orte_init followed by calls to
grpcomm.barrier
This is all completely benign unless actively selected. At the moment, it only supports modex-less launch for openib-based systems. Minor mod to the TCP btl would be required to enable it as well, if people are interested. Similarly, anyone interested in enabling other BTL's for modex-less operation should let me know and I'll give you the magic details.
This seems to significantly improve scalability provided the file can be locally located on the nodes. I'm looking at an alternative means of disseminating the info (perhaps in launch message) as an option for removing that constraint.
This commit was SVN r20098.
of more than one of the btl_openib_if_include, btl_openib_if_exclude,
btl_openib_ipaddr_include, or btl_openib_ipaddr_exclude MCA parameters.
This commit was SVN r20053.
determining which IP address to use when transmitting data. Also it adds logic
to prevent usage of more than one of the btl_openib_if_include,
btl_openib_if_exclude, btl_openib_ipaddr_include, or btl_openib_ipaddr_exclude
MCA parameters.
This should complete the code modifications needed for ticket 1665.
This commit was SVN r20052.
determing of the IP subnet. The netmask was being used improperly when
determining which subnet each connection is on. Part two is the ability to
include/exclude specific subnets.
This patch fixes ticket #1665
This commit was SVN r20016.
* If max_inline_data == -1 perform runtime detection
* If max_inline_data >=0 use the value provided
* If the user does not explicitly set this via command line, use the value from INI file
This commit fixes trac:1662
This commit was SVN r19995.
The following Trac tickets were found above:
Ticket 1662 --> https://svn.open-mpi.org/trac/ompi/ticket/1662
This will sit in trunk for a few days - would like to actually see some errors reported to syslog before moving the code to 1.3
This commit was SVN r19986.
another bug. This also causes 0-byte requests to be treated as a buffer
error, causing the base request to be requeued. On Cray XT, it may be
temporarily impossible to make allocations for buffer requests, as the
default stack size is small (8 MB) and there is no true swap device.
Even with the stack size increased, there will be cases in which this
condition recurs.
One possibility is to make the buffer allocations off of the heap; but,
this does not change the fact that eventually an out-of-memory condition
will occur and we need to support multiple receives in transit, a
condition for which the available buffer space may change. On the other
hand, if we switch to allocating the buffer space from the heap, we will
need to return an error when the allocation fails and there are no other
buffers in transit.
This commit was SVN r19981.
parameters on the connecting side also. Also move define of IF_NAMESIZE
into if.h file. And lastly, add one verbose debug message which may be
useful if we run into other issues like this.
This commit fixes trac:1573.
This commit was SVN r19932.
The following Trac tickets were found above:
Ticket 1573 --> https://svn.open-mpi.org/trac/ompi/ticket/1573
in case of failure more user friendly. In addition we now fully integrate
with the MX_BONDING features (i.e the multi-rails bonding can be done either
at the PML OB1 level, or deep down in the MX library). This feature is driven
by the btl_mx_bonding MCA parameter.
Plus some extra features:
- Correctly compute the bandwidth in case of bonding.
- Nicely handle any mapper access errors.
This commit was SVN r19874.
<infiniband/driver.h> cannot be included because it will fail to
compile. So per advice from Roland, in that case, just put manually
include the one ibv_*() prototype that we need. Use an undocumented
feature of AC_CHECK_HEADERS to check for <infiniband/driver.h> that I
was told on the autoconf mailing list (see the comment for more
details).
This commit was SVN r19857.
We were only allocating one whereas we would typically need
one per interface. Most of the time we got lucky and we
happily trundled over unallocated memory. But with enough
interfaces, we would get a SEGV.
Also add include to fix compilation with static libs.
This commit was SVN r19845.
THREAD_MULTIPLE. There's a new (hidden) MCA parameter to re-enable
these BTLs in the presence of THREAD_MULTIPLE:
btl_base_thread_multiple_override. This MCA parameter should ''only''
be used by developers who are working on make their BTLs thread safe;
it should ''not'' be used by end-users!
This commit was SVN r19826.
The following Trac tickets were found above:
Ticket 1588 --> https://svn.open-mpi.org/trac/ompi/ticket/1588
We had a public symbol named "already_opened". This commit changes
the name to mca_btl_base_already_opened.
I guess it's good we have the visibility stuff enabled by default!
:-)
This commit was SVN r19791.
1. Allow MX bonding via btl_mx_bonding MCA parameter. With this on, Open MPI
suppose that lib MX will do the bonding, and we will only return one BTL.
Otherwise, we return as many as devices.
2. Decrease the memory footprint, by cleaning up what we store about the
peers and how we store it.
3. Allow multiple MX routes that share the same mapper. In this particular
case we will link by their nic_id.
4. Allow multiple MX routes with multiple mappers. In this case we match
the NICs based on the last 6 digits of the mapper MAC.
5. Increase the size of the eager and rendez-vous eager limits in the
case where we are unable to register an unexpected callback with MX.
6. Increase the default max number of MX fragments.
7. Increase the max number of MX BTLs.
8. Only allow mx_if_include and mx_if_exclude if we have acess to the
mapper.
This commit was SVN r19788.
There is still a problem with OpenIB and threads (external to C/R functionality). It has been reported in Ticket #1539
Additionally:
* Fix a file cleanup bug in CRS Base.
* Fix a possible deadlock in the TCP ft_event function
* Add a mca_base_param_deregister() function to MCA base
* Add whole process checkpoint timers
* Add support for BTL: OpenIB, MX, Shared Memory
* Add support Mpool: rdma, sm
* Sundry bounds checking an cleanup in some scattered functions
This commit was SVN r19756.
I run IMB exchange on two QS22 machines with r19674 and it got stucked after 256 or 512 bytes every time.
After applying r19717 the test passed, so I guess this is a essential patch.
This commit was SVN r19752.
The following SVN revision numbers were found above:
r19674 --> open-mpi/ompi@15c47a2473
r19717 --> open-mpi/ompi@0a765cd788
following names are all new for v1.3, and therefore haven't been
officially released yet:
* btl_openib_of_cq_size
* btl_openib_of_max_inline_data
* btl_openib_of_pkey
* btl_openib_of_psn
* btl_openib_of_mtu
The "_of_" (for OpenFabrics) in there is redundant. It used to be
"_ib_", indicating that these values are pretty much passed directly
to the verbs stack. But I think the "openib" in the name implies this
already; having "_of_" in there just seems redundant, makes the name
longer, and seems redundant. It's also redundant.
So I took those "_of_"'s out of the MCA names. The old (v1.2) names
are still valid (but deprecated), such ash btl_openib_ib_cq_size.
This commit was SVN r19718.
* Change name: mca_btl_openib_of_pkey_value -> mca_btl_openib_of_pkey
(since now there's no index, the "_value" suffix is somewhat
superfluous)
* Put in a better help message for the _pkey MCA param (to agree with
the new help message in v1.2.8)
This commit was SVN r19716.
Commit from a long-standing Mercurial tree that ended up incorporating a lot of things:
* A few fixes for CPC interface changes in all the CPCs
* Attempts (but not yet finished) to fix shutdown problems in the IB CM CPC
* #1319: add CTS support (i.e., initiator guarantees to send first message; automatically activated for iWARP over the RDMA CM CPC)
* Some variable and function renamings to make this be generic (e.g., alloc_credit_frag became alloc_control_frag)
* CPCs no longer post receive buffers; they only post a single receive buffer for the CTS if they use CTS. Instead, the main BTL now posts the main sets of receive buffers.
* CPCs allocate a CTS buffer only if they're about to make a connection
* RDMA CM improvements:
* Use threaded mode openib fd monitoring to wait for for RDMA CM events
* Synchronize endpoint finalization and disconnection between main thread and service thread to avoid/fix some race conditions
* Converted several structs to be OBJs so that we can use reference counting to know when to invoke destructors
* Make some new OBJ's have opal_list_item_t's as their base, thereby eliminating the need for the local list_item_t type
* Renamed many variables to be internally consistent
* Centralize the decision in an inline function as to whether this process or the remote process is supposed to be the initiator
* Add oodles of OPAL_OUTPUT statements for debugging (hard-wired to output stream -1; to be activated by developers if they want/need them)
* Use rdma_create_qp() instead of ibv_create_qp()
* openib fd monitoring improvements:
* Renamed a bunch of functions and variables to be a little more obvious as to their true function
* Use pipes to communicate between main thread and service thread
* Add ability for main thread to invoke a function back on the service thread
* Ensure to set initiator_depth and responder_resources properly, but putting max_qp_rd_ataom and ma_qp_init_rd_atom in the modex (see rdma_connect(3))
* Ensure to set the source IP address in rdma_resolve() to ensure that we select the correct OpenFabrics source port
* Make new MCA param: openib_btl_connect_rdmacm_resolve_timeout
* Other improvements:
* btl_openib_device_type MCA param: can be "iw" or "ib" or "all" (or "infiniband" or "iwarp")
* Somewhat improved error handling
* Bunches of spelling fixes in comments, VERBOSE, and OUTPUT statements
* Oodles of little coding style fixes
* Changed shutdown ordering of btl; the device is now an OBJ with ref counting for destruction
* Added some more show_help error messages
* Change configury to only build IBCM / RDMACM if we have threads (because we need a progress thread)
This commit was SVN r19686.
The following Trac tickets were found above:
Ticket 1210 --> https://svn.open-mpi.org/trac/ompi/ticket/1210
(related to the presence of posix threads and ptmalloc2) is now a
little outdated: since we don't build ptmalloc2 as part of libopal
anymore, the openib BTL's requirements are not directly tied to
ptmalloc2's anymore. Specifically, I altered the test to:
1. At compile time, if no threads are found, the ptmalloc2 component
is going to be built, '''and the ptmalloc2 component is going to be
inside libopal,''' then refuse to build the openib BTL.
1. At run time, if no threads were available at compile time and the
ptmalloc2 component is part of the process, then refuse to use the
openib BTL.
Fixes trac:1537.
This commit was SVN r19652.
The following Trac tickets were found above:
Ticket 1537 --> https://svn.open-mpi.org/trac/ompi/ticket/1537
figure it out at runtime (really meaning: we'll still default to "0"
unless something explicitly overrides to 1, such as the openib BTL).
This way, ompi_info doesn't confusingly report mpi_leave_pinned==0 for
mpi_leave_pinned, but we end up running with mpi_leave_pinned==1.
Fixes trac:1502.
This commit was SVN r19571.
The following Trac tickets were found above:
Ticket 1502 --> https://svn.open-mpi.org/trac/ompi/ticket/1502
mostly don't use this mechanism, as we have to be thread safe in order to
really take full advantage of it. The unexpected handler is called by one
of th MX threads, and we do not have control on the moment. In a non threaded
case, this will completely destroy our recv requests queues, so the safest
approach is to don't the unexpected handler.
This commit was SVN r19496.
See opal/mca/paffinity/paffinity.h for explanation as to the physical vs logical nature of the params used in the API.
Fixes trac:1435
This commit was SVN r19391.
The following Trac tickets were found above:
Ticket 1435 --> https://svn.open-mpi.org/trac/ompi/ticket/1435
Roland noticed that the QLogic HCA driver was using the PCIe vendor ID
for the ibv_query_device so the IEEE OUI value is now used. This means
the config file should recognize the vendor ID value 0x1175 too.
This commit was SVN r19277.
This is setup so that it only is issued once (as opposed to every time they do it), and goes through orte_show_help so the user doesn't get hammered by #procs copies of the warning. In addition, there is a new MCA param (can't have too many!) to shut the warning off altogether.
This closes ticket #1244
This commit was SVN r19196.
* add "register" function to mca_base_component_t
* converted coll:basic and paffinity:linux and paffinity:solaris to
use this function
* we'll convert the rest over time (I'll file a ticket once all
this is committed)
* add 32 bytes of "reserved" space to the end of mca_base_component_t
and mca_base_component_data_2_0_0_t to make future upgrades
[slightly] easier
* new mca_base_component_t size: 196 bytes
* new mca_base_component_data_2_0_0_t size: 36 bytes
* MCA base version bumped to v2.0
* '''We now refuse to load components that are not MCA v2.0.x'''
* all MCA frameworks versions bumped to v2.0
* be a little more explicit about version numbers in the MCA base
* add big comment in mca.h about versioning philosophy
This commit was SVN r19073.
The following Trac tickets were found above:
Ticket 1392 --> https://svn.open-mpi.org/trac/ompi/ticket/1392
* Use synonym/deprecated MCA param API for some mca base params
* In openib BTL, if we have appropriate memory hooks support, and if
mpi_leave_pinned and mpi_leave_pinned_pipeline were not set by the
user, set mpi_leave_pinned to 1.
* Defer checking mpi_leave_pinned_* until as late as possible (i.e.,
until after the btl's have had a chance to set mpi_leave_pinned to
1):
* in ob1 pml
* in rdma mpool
This commit was SVN r19022.
The following Trac tickets were found above:
Ticket 1379 --> https://svn.open-mpi.org/trac/ompi/ticket/1379
"!OpenFabrics" / neutral (i.e., refer to IB and/or iWARP).
* Mostly just type, variable/field, and funcion name changes, such as
s/hca/device/g, etc.
* Changed the INI file for the hardware-specific parameters to be
mca-btl-openib-device-params.ini.
* Updated a lot of help messages in the help-*.txt files, not just to
update it to be !OpenFabrics/neutral language, but also for some
consistency of tone, indenting, etc.
* Deprecated a bunch of MCA params in favor of language-neutral new
ones:
* btl_openib_warn_no_hca_params_found (s/hca/device/)
* btl_openib_hca_param_files
* btl_openib_ib_cq_size (s/_ib_/_of_/)
* btl_openib_ib_max_inline_data
* btl_openib_ib_psn
* btl_openib_ib_mtu
* btl_openib_ib_pkey_ix
* btl_openib_ib_pkey_val
This commit was SVN r18985.
The following Trac tickets were found above:
Ticket 1295 --> https://svn.open-mpi.org/trac/ompi/ticket/1295
The rdmacm event handler has no way of reporting fatal errors to the upper
layers. By calling mca_btl_openib_endpoint_invoke_error in the rdmacm event
handler for the errors encountered, these errors can now be handled
appropriately.
Closes out Ticket #1283
This commit was SVN r18980.
If IBCM was explicitly specified with exclude/include parameter,
OpenIB BTL will enable verbose report for "/dev/infiniband/ucm" error,
other way the error will not be reported.
This commit was SVN r18868.
first to the trunk. So, here is the trunk checkin:
The call to orte_show_help() to notify truncation of the max_inline value
was missing the want_error_header boolean, which eventually results in
a SEGV. This change corrects the call with the bool set to true.
This commit was SVN r18839.
* Move the passive side QP move to RTS to before we send the reply
(vs. sending it after we get the RTU). A lengthy comment explains
the need for this.
* Add some timers to the code for analyzing where time is spent.
* Clarify a few error messages.
* Currently have a loop around ib_cm_listen() because sometimes it
fails for seeminly no reason. Have pending e-mails in to Sean
Hefty to see if we can figure out why this is happening. Note that
the more MPI processes you add, the more likely this error is to
occur (e.g., ran 720 processes and it happens at least 50% of the
time). This makes IBCM somewhat unattractive for general use;
hopefully we can get a fix...
This commit was SVN r18766.
endpoint.c) because it's almost identical in all the CPC's. OOB,
XOOB, and IBCM now all invoke the btl error handler properly if
there's an error during wireup. RDMACM still needs to be done.
This commit was SVN r18764.
The following SVN revision numbers were found above:
r18762 --> open-mpi/ompi@3eda04578f
upper level btl (and therefore the PML) when something goes wrong
during wireup.
Refs trac:1283.
This commit was SVN r18762.
The following Trac tickets were found above:
Ticket 1283 --> https://svn.open-mpi.org/trac/ompi/ticket/1283
* Properly handle non-symmetric subnet ID's
* Be a bit more stringent when checking for the GID
* Add lots of BTL_VERBOSE's for diagnostics
This commit was SVN r18754.
as the send and the put share the same execution path and in the case of the
put the MCA_BTL_DES_SEND_ALWAYS_CALLBACK was not set.
This commit was SVN r18735.
ompi_info will crash due to a reference to a unalloced struct. Add a check for
this struct to prevent this crash.
This commit was SVN r18727.
The following SVN revision numbers were found above:
r18725 --> open-mpi/ompi@c2351d39f1
If rdmacm cpc is disabled and an iwarp device is forced to be used, then
mca_btl_openib_get_iwarp_subnet_id will attempt to reference an uninitalized
list. Modify the code to check for the uninitalized list and return zero.
Also, fix a memory leak caused by not freeing alloced addresses structs during
shutdown.
This commit was SVN r18725.
the fragments that failed to be send, there is no need to replicate the same
mechanism in the BTL.
Force the SM BTL to empty all ack fragments in the component progress function.
This commit was SVN r18724.
INI file use_eager_rdma value (fixes trac:1169)
* fixed a typo in a MCA param help message
* made the check for enabling short/eager RDMA more robust in the
presence of progress threads; it now emits a show_help warning
This commit was SVN r18723.
The following Trac tickets were found above:
Ticket 1169 --> https://svn.open-mpi.org/trac/ompi/ticket/1169
specified, probe for max value supported by device.
This commit was SVN r18720.
The following Trac tickets were found above:
Ticket 1355 --> https://svn.open-mpi.org/trac/ompi/ticket/1355
* Ensure to destroy the correct QP (local->id[num]->qp will always
have a valid pointer in it, even if we setup a dummy qp)
* Note two notable places where we need to figure out how to
propagate errors up from the CPC to the main BTL / PML when errors
occur. Probably have the same issue in IBCM, too.
This commit was SVN r18700.
two processes on the same server (!). So for today, we'll simply mark
all local processes that use iWARP adapters as "unreachable".
More details in #1352.
This commit was SVN r18699.
(e.g., if you're excluding some devices, their destructors will be
invoked before the async event thread was setup for them).
This commit was SVN r18698.
multiple adapters (eg., Chelsio T3).
But we need to figure out how to determine a good value for the
resident adapter(s) at runtime. It's problematic because, for
example, Mellanox ConnectX and Chelsio T3 report max_inline values
differently at run-time. If you ibv_create_qp with a max_inline value
of 0, ConnectX reports back a value that is a formular based on a few
other values (e.g., max_send_sge and max_recv_sge). But T3 always
reports back "64".
We're looking into this to figure out the best way -- reducing the
default right now should allow other adapters to run while we figure
it out.
This commit was SVN r18697.
After much work by Jeff and myself, and quite a lot of discussion, it has become clear that we simply cannot resolve the infinite loops caused by RML-involved subsystems calling orte_output. The original rationale for the change to orte_output has also been reduced by shifting the output of XML-formatted vs human readable messages to an alternative approach.
I have globally replaced the orte_output/ORTE_OUTPUT calls in the code base, as well as the corresponding .h file name. I have test compiled and run this on the various environments within my reach, so hopefully this will prove minimally disruptive.
This commit was SVN r18619.
We already show_help when we fail to create queues, so I just made the
message a little more verbose such that it may be that OMPI is trying
to use a feature that is not supported on the hardware.
This commit was SVN r18553.
The following Trac tickets were found above:
Ticket 1121 --> https://svn.open-mpi.org/trac/ompi/ticket/1121
1. The send path get shorter. The BTL is allowed to return > 0 to specify that the
descriptor was pushed to the networks, and that the memory attached to it is
available again for the upper layer. The MCA_BTL_DES_SEND_ALWAYS_CALLBACK flag
can be used by the PML to force the BTL to always trigger the callback.
Unmodified BTL will continue to work as expected, as they will return OMPI_SUCCESS
which force the PML to have exactly the same behavior as before. Some BTLs have
been modified: self, sm, tcp, mx.
2. Add send immediate interface to BTL.
The idea is to have a mechanism of allowing the BTL to take advantage of
send optimizations such as the ability to deliver data "inline". Some
network APIs such as Portals allow data to be sent using a "thin" event
without packing data into a memory descriptor. This interface change
allows the BTL to use such capabilities and allows for other optimizations
in the future. All existing BTLs except for Portals and sm have this interface
set to NULL.
This commit was SVN r18551.
that it's a directory. That's good enough to know that the
OpenFabrics kernel drivers have been loaded. If you have no RDMA
devices and don't want to see the OMPI warning about not finding any
devices, then don't start the OpenFabrics kernel drivers.
This commit was SVN r18540.
non-empty. If not, then exit the openib btl silently. This addresses
the case where libibverbs is installed (which is getting more common)
and therefore the openib BTL was built/installed, but the kernel
drivers are not loaded (assumedly because there is no RDMA hardware
present). In this case, "mpirun a.out" will not issue a warning.
There appears to be no good way to definitely tell if there are no
RDMA hardware devices present. For example, if libibverbs/the openib
BTL is installed, there are no RDMA devices present, but the RDMA
hardware kernel drivers ''are'' loaded, OMPI will warn that it was
unable to find suitable devices. This warning is easily eliminated by
unloading the kernel drivers.
This commit was SVN r18530.
The following Trac tickets were found above:
Ticket 1305 --> https://svn.open-mpi.org/trac/ompi/ticket/1305
