There is still a problem with OpenIB and threads (external to C/R functionality). It has been reported in Ticket #1539
Additionally:
* Fix a file cleanup bug in CRS Base.
* Fix a possible deadlock in the TCP ft_event function
* Add a mca_base_param_deregister() function to MCA base
* Add whole process checkpoint timers
* Add support for BTL: OpenIB, MX, Shared Memory
* Add support Mpool: rdma, sm
* Sundry bounds checking an cleanup in some scattered functions
This commit was SVN r19756.
I run IMB exchange on two QS22 machines with r19674 and it got stucked after 256 or 512 bytes every time.
After applying r19717 the test passed, so I guess this is a essential patch.
This commit was SVN r19752.
The following SVN revision numbers were found above:
r19674 --> open-mpi/ompi@15c47a2473
r19717 --> open-mpi/ompi@0a765cd788
We still do an interreduce but it is now followed by an intrascatterv.
This fixes trac:1554.
This commit was SVN r19723.
The following Trac tickets were found above:
Ticket 1554 --> https://svn.open-mpi.org/trac/ompi/ticket/1554
following names are all new for v1.3, and therefore haven't been
officially released yet:
* btl_openib_of_cq_size
* btl_openib_of_max_inline_data
* btl_openib_of_pkey
* btl_openib_of_psn
* btl_openib_of_mtu
The "_of_" (for OpenFabrics) in there is redundant. It used to be
"_ib_", indicating that these values are pretty much passed directly
to the verbs stack. But I think the "openib" in the name implies this
already; having "_of_" in there just seems redundant, makes the name
longer, and seems redundant. It's also redundant.
So I took those "_of_"'s out of the MCA names. The old (v1.2) names
are still valid (but deprecated), such ash btl_openib_ib_cq_size.
This commit was SVN r19718.
* Change name: mca_btl_openib_of_pkey_value -> mca_btl_openib_of_pkey
(since now there's no index, the "_value" suffix is somewhat
superfluous)
* Put in a better help message for the _pkey MCA param (to agree with
the new help message in v1.2.8)
This commit was SVN r19716.
Commit from a long-standing Mercurial tree that ended up incorporating a lot of things:
* A few fixes for CPC interface changes in all the CPCs
* Attempts (but not yet finished) to fix shutdown problems in the IB CM CPC
* #1319: add CTS support (i.e., initiator guarantees to send first message; automatically activated for iWARP over the RDMA CM CPC)
* Some variable and function renamings to make this be generic (e.g., alloc_credit_frag became alloc_control_frag)
* CPCs no longer post receive buffers; they only post a single receive buffer for the CTS if they use CTS. Instead, the main BTL now posts the main sets of receive buffers.
* CPCs allocate a CTS buffer only if they're about to make a connection
* RDMA CM improvements:
* Use threaded mode openib fd monitoring to wait for for RDMA CM events
* Synchronize endpoint finalization and disconnection between main thread and service thread to avoid/fix some race conditions
* Converted several structs to be OBJs so that we can use reference counting to know when to invoke destructors
* Make some new OBJ's have opal_list_item_t's as their base, thereby eliminating the need for the local list_item_t type
* Renamed many variables to be internally consistent
* Centralize the decision in an inline function as to whether this process or the remote process is supposed to be the initiator
* Add oodles of OPAL_OUTPUT statements for debugging (hard-wired to output stream -1; to be activated by developers if they want/need them)
* Use rdma_create_qp() instead of ibv_create_qp()
* openib fd monitoring improvements:
* Renamed a bunch of functions and variables to be a little more obvious as to their true function
* Use pipes to communicate between main thread and service thread
* Add ability for main thread to invoke a function back on the service thread
* Ensure to set initiator_depth and responder_resources properly, but putting max_qp_rd_ataom and ma_qp_init_rd_atom in the modex (see rdma_connect(3))
* Ensure to set the source IP address in rdma_resolve() to ensure that we select the correct OpenFabrics source port
* Make new MCA param: openib_btl_connect_rdmacm_resolve_timeout
* Other improvements:
* btl_openib_device_type MCA param: can be "iw" or "ib" or "all" (or "infiniband" or "iwarp")
* Somewhat improved error handling
* Bunches of spelling fixes in comments, VERBOSE, and OUTPUT statements
* Oodles of little coding style fixes
* Changed shutdown ordering of btl; the device is now an OBJ with ref counting for destruction
* Added some more show_help error messages
* Change configury to only build IBCM / RDMACM if we have threads (because we need a progress thread)
This commit was SVN r19686.
The following Trac tickets were found above:
Ticket 1210 --> https://svn.open-mpi.org/trac/ompi/ticket/1210
got a whole lot smaller, decreasing the memory footprint of the
running application. How much it's a good question. Here is a
breakdown:
- in mca_bml_base_endpoint_t: 3 *size_t + 1 * uint32_t
- in mca_bml_base_btl_t: 1 * int + 1 * double - 1 * float
+ 6 * size_t + 9 * (void*)
The decrease in mca_bml_base_endpoint_t is for each peer and the
decrease in mca_bml_base_btl_t is for each BTL for each peer.
So, if we consider the most convenient case where there is only
one network between all peers, this decrease the memory foot print
per peer by
9*size_t + 9*(void*) + 2 * int32_t + 1 * double - 1 * float.
On a 64 bits machine this will be 156 bytes per peer.
Now we access all these fields directly from the underlying BTL
structure, and as this structure is common to multiple BML endpoint,
we are a lot more cache friendly. Even if this do not improve the
latency, it makes the SM performance graph a lot smoother.
This commit was SVN r19659.
There was an argument that was barely used, and on return at the PML
level it contained nothing usable. It has been removed, so now we're
using less memory ...
This commit was SVN r19657.
(related to the presence of posix threads and ptmalloc2) is now a
little outdated: since we don't build ptmalloc2 as part of libopal
anymore, the openib BTL's requirements are not directly tied to
ptmalloc2's anymore. Specifically, I altered the test to:
1. At compile time, if no threads are found, the ptmalloc2 component
is going to be built, '''and the ptmalloc2 component is going to be
inside libopal,''' then refuse to build the openib BTL.
1. At run time, if no threads were available at compile time and the
ptmalloc2 component is part of the process, then refuse to use the
openib BTL.
Fixes trac:1537.
This commit was SVN r19652.
The following Trac tickets were found above:
Ticket 1537 --> https://svn.open-mpi.org/trac/ompi/ticket/1537
Thanks to George and Jeff for pointing out a better way to do this.
This commit was SVN r19573.
The following SVN revision numbers were found above:
r19566 --> open-mpi/ompi@351c3a3a86
figure it out at runtime (really meaning: we'll still default to "0"
unless something explicitly overrides to 1, such as the openib BTL).
This way, ompi_info doesn't confusingly report mpi_leave_pinned==0 for
mpi_leave_pinned, but we end up running with mpi_leave_pinned==1.
Fixes trac:1502.
This commit was SVN r19571.
The following Trac tickets were found above:
Ticket 1502 --> https://svn.open-mpi.org/trac/ompi/ticket/1502
remove the unconditional opal_output's when mmap() fails, and instead,
conditionally output the failure message via btl_base_verbose settings.
This commit was SVN r19547.
The new component fixes a number of problems with the old component. The core algorithm is the same, but by changing the data strucutres a bit we have improved performance and memory utilization.
There are still a couple corner cases that still need some work. However, I did not want to delay bringing this into the trunk (and v1.3 branch) for too much longer.
This commit was SVN r19537.
checking for contiguous datatypes by using our native DDT engine
(rather than several MPI_* calls). The majority of the work is in the
IO ROMIO module.c file, but there's a small part in
adio/common/iscontig.c that we're also submitting upstream.
This commit was SVN r19509.
mostly don't use this mechanism, as we have to be thread safe in order to
really take full advantage of it. The unexpected handler is called by one
of th MX threads, and we do not have control on the moment. In a non threaded
case, this will completely destroy our recv requests queues, so the safest
approach is to don't the unexpected handler.
This commit was SVN r19496.
way".
Don't modify coords in the top-level API function because coords is an
IN variable. Instead, as Nysal noted, the real cause of the problem
was a missing ! down in topo_base_cart_rank.c. Put a comment down in
topo_base_cart_rank.c explaining what's going on so that the code is
not so cryptic.
Refs trac:1363.
This commit was SVN r19487.
The following Trac tickets were found above:
Ticket 1363 --> https://svn.open-mpi.org/trac/ompi/ticket/1363
* Various changes to enable 0-dimensional cartesian communicators:
* Set various mtc_* members to NULL when there are 0 dimensions (and
don't bother trying to memcpy these arrays when duplicating the
communicator -- because they're NULL)
* adjust topo_base_cart_sub to correctly handle 0 dimensions
(simplified it a bit)
* adjust a few error codes to return ERR_OUT_OF_RESOURCE
* adjust error checking of CART_CREATE, CART_RANK
* Allow MPI_GRAPH_CREATE to accept 0 == nnodes.
* Bump reported MPI version in mpi.h to 2.1
This commit was SVN r19461.
The following Trac tickets were found above:
Ticket 1236 --> https://svn.open-mpi.org/trac/ompi/ticket/1236
guarantee local progress, my opinion is we don't need to do anything
special for complete sends. Perhaps I'm wrong on that one, in which
case this implementation of send_complete is not quite right.
* Give a slightly better error message when a catchall is hit, so that
we can at least know *which* catchall is hit.
This commit was SVN r19456.
See opal/mca/paffinity/paffinity.h for explanation as to the physical vs logical nature of the params used in the API.
Fixes trac:1435
This commit was SVN r19391.
The following Trac tickets were found above:
Ticket 1435 --> https://svn.open-mpi.org/trac/ompi/ticket/1435
2. fix a bug in the initial communicator creation of llcomm
3. fix a bug which showed up as the result of fixing issue number 2: we have
to check now whether llcomm has really be created before freeing the
according llcomm in hierarch_destruct.
This commit was SVN r19361.
generate an overlap between the different layers. Why four versions? Because
there is right now always the trade-off between using non-blocking operations
on a layer with a trivial, linear algorithm and using the more sophisticaed
algorithms in a blocking manner.
- bcast_intra_seg used the bcast of lcomm and llcomm, similarly
to original algorithm in hierarch. However, it can segment
the message, such that we might get an overlap between the two
layers. This overlap is based on the assumption, that a process
might be done early with a bcast and can start the next one.
- bcast_intra_seg1: replaces the llcomm->bcast by isend/irecvs
to increase the overlap, keeps the lcomm->bcast however
- bcast_intra_seg2: replaced lcomm->bcast by isend/irecvs
to increase the overlap, keeps however llcomm->bcast
- bcast_intra_seg3: replaced both lcomm->bcast and llcomm->bcast
by isend/irecvs
The code is lightly tested, more testing to follow right now.
This commit was SVN r19358.
Roland noticed that the QLogic HCA driver was using the PCIe vendor ID
for the ibv_query_device so the IEEE OUI value is now used. This means
the config file should recognize the vendor ID value 0x1175 too.
This commit was SVN r19277.
* Make the creation of the build dir for the man pages a bit more
robust (thanks to suggestions from Ralf W.).
* Only distribute the .Xin files, not the .X man pages themselves.
* Make the .X files depend on opal_config.h so that if you re-run
configure and change opal_config.h (e.g., a new version), the man
pages should get rebuilt.
* Man pages are now cleaned with "distclean", not "maintainer-clean".
* Fix a typo in opal_crs.7in.
* Udpate make_dist_tarball to update "date" in the VERSION file.
* Make make_dist_tarball a bit friendlier to hg checkouts.
This commit was SVN r19219.
This is setup so that it only is issued once (as opposed to every time they do it), and goes through orte_show_help so the user doesn't get hammered by #procs copies of the warning. In addition, there is a new MCA param (can't have too many!) to shut the warning off altogether.
This closes ticket #1244
This commit was SVN r19196.
(it is just set within MCA_PML_OB1_RECV_REQUEST_UNPACK)
Iff Coverity's prevent makes usage of __attribute__(unused),
this should get rid of warning.
Relates to CID1060
Would then apply to a many int _rc; definitions, that are
used in other macros in similar fashion...
This commit was SVN r19179.
Revise the scope precedence in the MPI_Publish, Unpublish, and Lookup functions. If a global server was specified and is available, then default to using it for all three functions. If not, then default to using local scope.
If an info_key was provided, then it takes preference. We always follow the user's direction - this change only impacts the scope ordering if the user -doesn't- tell us the order to use.
This commit was SVN r19146.
versions, dates and build names.
Fixes trac:1387
Big thanks to Jeff and Brian for help and oversight.
This commit was SVN r19120.
The following Trac tickets were found above:
Ticket 1387 --> https://svn.open-mpi.org/trac/ompi/ticket/1387
that by decreasing the btl_inuse if there was already a registered BTL we basically reset
the changes for this new BTL to register it's progress function, even if it was supposed to
handle another peer.
This commit was SVN r19080.
* add "register" function to mca_base_component_t
* converted coll:basic and paffinity:linux and paffinity:solaris to
use this function
* we'll convert the rest over time (I'll file a ticket once all
this is committed)
* add 32 bytes of "reserved" space to the end of mca_base_component_t
and mca_base_component_data_2_0_0_t to make future upgrades
[slightly] easier
* new mca_base_component_t size: 196 bytes
* new mca_base_component_data_2_0_0_t size: 36 bytes
* MCA base version bumped to v2.0
* '''We now refuse to load components that are not MCA v2.0.x'''
* all MCA frameworks versions bumped to v2.0
* be a little more explicit about version numbers in the MCA base
* add big comment in mca.h about versioning philosophy
This commit was SVN r19073.
The following Trac tickets were found above:
Ticket 1392 --> https://svn.open-mpi.org/trac/ompi/ticket/1392
have/need lseek64" issue. This fix is also included in MPICH2 after
v1.0.7.
This commit was SVN r19070.
The following Trac tickets were found above:
Ticket 1419 --> https://svn.open-mpi.org/trac/ompi/ticket/1419
SVN version (r19045), but I also edited the svn:ignore to ignore these
files in the same SVN commit -- I suspect that SVN got confused and
did not actually delete them.
This commit was SVN r19048.
The following SVN revision numbers were found above:
r19045 --> open-mpi/ompi@63b63d48c3
Update the version of ROMIO to that which was contained in
MPICH2-1.0.7, plus a few patches from the upstream ROMIO maintainers
(because OMPI uses a few code paths in ROMIO that MPICH2 does not;
there were a few compile bugs in the ROMIO from MPICH2-1.0.7).
Added an info MCA param to be able to tell which version of ROMIO is
contained in OMPI: io_romio_version.
Many, many thanks to romio-maint@mcs.anl.gov for all their help in
integrating this new version of ROMIO into Open MPI.
This commit was SVN r19045.
The following Trac tickets were found above:
Ticket 1370 --> https://svn.open-mpi.org/trac/ompi/ticket/1370
* Use synonym/deprecated MCA param API for some mca base params
* In openib BTL, if we have appropriate memory hooks support, and if
mpi_leave_pinned and mpi_leave_pinned_pipeline were not set by the
user, set mpi_leave_pinned to 1.
* Defer checking mpi_leave_pinned_* until as late as possible (i.e.,
until after the btl's have had a chance to set mpi_leave_pinned to
1):
* in ob1 pml
* in rdma mpool
This commit was SVN r19022.
The following Trac tickets were found above:
Ticket 1379 --> https://svn.open-mpi.org/trac/ompi/ticket/1379
a little bit more than "BTL was able to add some procs". The real condition to
allow the BTL progress is that we will use it to send/recv data to/from some
of the peers (this include the BTL exclusivity in the process).
This commit was SVN r19010.
"!OpenFabrics" / neutral (i.e., refer to IB and/or iWARP).
* Mostly just type, variable/field, and funcion name changes, such as
s/hca/device/g, etc.
* Changed the INI file for the hardware-specific parameters to be
mca-btl-openib-device-params.ini.
* Updated a lot of help messages in the help-*.txt files, not just to
update it to be !OpenFabrics/neutral language, but also for some
consistency of tone, indenting, etc.
* Deprecated a bunch of MCA params in favor of language-neutral new
ones:
* btl_openib_warn_no_hca_params_found (s/hca/device/)
* btl_openib_hca_param_files
* btl_openib_ib_cq_size (s/_ib_/_of_/)
* btl_openib_ib_max_inline_data
* btl_openib_ib_psn
* btl_openib_ib_mtu
* btl_openib_ib_pkey_ix
* btl_openib_ib_pkey_val
This commit was SVN r18985.
The following Trac tickets were found above:
Ticket 1295 --> https://svn.open-mpi.org/trac/ompi/ticket/1295
The rdmacm event handler has no way of reporting fatal errors to the upper
layers. By calling mca_btl_openib_endpoint_invoke_error in the rdmacm event
handler for the errors encountered, these errors can now be handled
appropriately.
Closes out Ticket #1283
This commit was SVN r18980.
If IBCM was explicitly specified with exclude/include parameter,
OpenIB BTL will enable verbose report for "/dev/infiniband/ucm" error,
other way the error will not be reported.
This commit was SVN r18868.
first to the trunk. So, here is the trunk checkin:
The call to orte_show_help() to notify truncation of the max_inline value
was missing the want_error_header boolean, which eventually results in
a SEGV. This change corrects the call with the bool set to true.
This commit was SVN r18839.
see how the next gen panasas stuff does in terms of warnings; we can
always re-merge this later if we want to. It's just easier if we have
as little OMPI-specific code as possible (particularly when we know
that the panasas code has some big changes coming).
This commit was SVN r18823.
The following SVN revision numbers were found above:
r17543 --> open-mpi/ompi@b4ec81a9fd
