2004-07-16 18:28:51 +04:00
|
|
|
#
|
2005-11-05 22:57:48 +03:00
|
|
|
# Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
|
|
|
|
# University Research and Technology
|
|
|
|
|
# Corporation. All rights reserved.
|
|
|
|
|
# Copyright (c) 2004-2005 The University of Tennessee and The University
|
|
|
|
|
# of Tennessee Research Foundation. All rights
|
|
|
|
|
# reserved.
|
2004-11-28 23:09:25 +03:00
|
|
|
# Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
|
|
|
# University of Stuttgart. All rights reserved.
|
2005-03-24 15:43:37 +03:00
|
|
|
# Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
|
# All rights reserved.
|
2007-03-01 16:39:20 +03:00
|
|
|
# Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
2004-11-22 04:38:40 +03:00
|
|
|
# $COPYRIGHT$
|
|
|
|
|
#
|
|
|
|
|
# Additional copyrights may follow
|
|
|
|
|
#
|
2004-07-16 18:28:51 +04:00
|
|
|
# $HEADER$
|
|
|
|
|
#
|
|
|
|
|
|
2008-01-02 21:00:20 +03:00
|
|
|
AM_CPPFLAGS = $(btl_openib_CPPFLAGS)
|
Bring over all the work from the /tmp/ib-hw-detect branch. In
addition to my design and testing, it was conceptually approved by
Gil, Gleb, Pasha, Brad, and Galen. Functionally [probably somewhat
lightly] tested by Galen. We may still have to shake out some bugs
during the next few months, but it seems to be working for all the
cases that I can throw at it.
Here's a summary of the changes from that branch:
* Move MCA parameter registration to a new file (btl_openib_mca.c):
* Properly check the retun status of registering MCA params
* Check for valid values of MCA parameters
* Make help strings better
* Otherwise, the only default value of an MCA param that was
changed was max_btls; it went from 4 to -1 (meaning: use all
available)
* Properly prototyped internal functions in _component.c
* Made a bunch of functions static that didn't need to be public
* Renamed to remove "mca_" prefix from static functions
* Call new MCA param registration function
* Call new INI file read/lookup/finalize functions
* Updated a bunch of macros to be "BTL_" instead of "ORTE_"
* Be a little more consistent with return values
* Handle -1 for the max_btls MCA param
* Fixed a free() that should have been an OBJ_RELEASE()
* Some re-indenting
* Added INI-file parsing
* New flex file: btl_openib_ini.l
* New default HCA params .ini file (probably to be expanded over
time by other HCA vendors)
* Added more show_help messages for parsing problems
* Read in INI files and cache the values for later lookup
* When component opens an HCA, lookup to see if any corresponding
values were found in the INI files (ID'ed by the HCA vendor_id
and vendor_part_id)
* Added btl_openib_verbose MCA param that shows what the INI-file
stuff does (e.g., shows which MTU your HCA ends up using)
* Added btl_openib_hca_param_files as a colon-delimited list of INI
files to check for values during startup (in order,
left-to-right, just like the MCA base directory param).
* MTU is currently the only value supported in this framework.
* It is not a fatal error if we don't find params for the HCA in
the INI file(s). Instead, just print a warning. New MCA param
btl_openib_warn_no_hca_params_found can be used to disable
printing the warning.
* Add MTU to peer negotiation when making a connection
* Exchange maximum MTU; select the lesser of the two
This commit was SVN r11182.
2006-08-14 23:30:37 +04:00
|
|
|
AM_LFLAGS = -Pbtl_openib_ini_yy
|
|
|
|
|
LEX_OUTPUT_ROOT = lex.btl_openib_ini_yy
|
2005-09-07 09:54:53 +04:00
|
|
|
|
2007-03-01 16:39:20 +03:00
|
|
|
amca_paramdir = $(AMCA_PARAM_SETS_DIR)
|
|
|
|
|
dist_amca_param_DATA = btl-openib-benchmark
|
|
|
|
|
|
Bring over all the work from the /tmp/ib-hw-detect branch. In
addition to my design and testing, it was conceptually approved by
Gil, Gleb, Pasha, Brad, and Galen. Functionally [probably somewhat
lightly] tested by Galen. We may still have to shake out some bugs
during the next few months, but it seems to be working for all the
cases that I can throw at it.
Here's a summary of the changes from that branch:
* Move MCA parameter registration to a new file (btl_openib_mca.c):
* Properly check the retun status of registering MCA params
* Check for valid values of MCA parameters
* Make help strings better
* Otherwise, the only default value of an MCA param that was
changed was max_btls; it went from 4 to -1 (meaning: use all
available)
* Properly prototyped internal functions in _component.c
* Made a bunch of functions static that didn't need to be public
* Renamed to remove "mca_" prefix from static functions
* Call new MCA param registration function
* Call new INI file read/lookup/finalize functions
* Updated a bunch of macros to be "BTL_" instead of "ORTE_"
* Be a little more consistent with return values
* Handle -1 for the max_btls MCA param
* Fixed a free() that should have been an OBJ_RELEASE()
* Some re-indenting
* Added INI-file parsing
* New flex file: btl_openib_ini.l
* New default HCA params .ini file (probably to be expanded over
time by other HCA vendors)
* Added more show_help messages for parsing problems
* Read in INI files and cache the values for later lookup
* When component opens an HCA, lookup to see if any corresponding
values were found in the INI files (ID'ed by the HCA vendor_id
and vendor_part_id)
* Added btl_openib_verbose MCA param that shows what the INI-file
stuff does (e.g., shows which MTU your HCA ends up using)
* Added btl_openib_hca_param_files as a colon-delimited list of INI
files to check for values during startup (in order,
left-to-right, just like the MCA base directory param).
* MTU is currently the only value supported in this framework.
* It is not a fatal error if we don't find params for the HCA in
the INI file(s). Instead, just print a warning. New MCA param
btl_openib_warn_no_hca_params_found can be used to disable
printing the warning.
* Add MTU to peer negotiation when making a connection
* Exchange maximum MTU; select the lesser of the two
This commit was SVN r11182.
2006-08-14 23:30:37 +04:00
|
|
|
dist_pkgdata_DATA = \
|
|
|
|
|
help-mpi-btl-openib.txt \
|
|
|
|
|
mca-btl-openib-hca-params.ini
|
2006-06-06 01:23:45 +04:00
|
|
|
|
2005-07-01 01:28:35 +04:00
|
|
|
sources = \
|
|
|
|
|
btl_openib.c \
|
|
|
|
|
btl_openib.h \
|
|
|
|
|
btl_openib_component.c \
|
|
|
|
|
btl_openib_endpoint.c \
|
|
|
|
|
btl_openib_endpoint.h \
|
|
|
|
|
btl_openib_frag.c \
|
|
|
|
|
btl_openib_frag.h \
|
|
|
|
|
btl_openib_proc.c \
|
2006-03-26 12:30:50 +04:00
|
|
|
btl_openib_proc.h \
|
Bring over all the work from the /tmp/ib-hw-detect branch. In
addition to my design and testing, it was conceptually approved by
Gil, Gleb, Pasha, Brad, and Galen. Functionally [probably somewhat
lightly] tested by Galen. We may still have to shake out some bugs
during the next few months, but it seems to be working for all the
cases that I can throw at it.
Here's a summary of the changes from that branch:
* Move MCA parameter registration to a new file (btl_openib_mca.c):
* Properly check the retun status of registering MCA params
* Check for valid values of MCA parameters
* Make help strings better
* Otherwise, the only default value of an MCA param that was
changed was max_btls; it went from 4 to -1 (meaning: use all
available)
* Properly prototyped internal functions in _component.c
* Made a bunch of functions static that didn't need to be public
* Renamed to remove "mca_" prefix from static functions
* Call new MCA param registration function
* Call new INI file read/lookup/finalize functions
* Updated a bunch of macros to be "BTL_" instead of "ORTE_"
* Be a little more consistent with return values
* Handle -1 for the max_btls MCA param
* Fixed a free() that should have been an OBJ_RELEASE()
* Some re-indenting
* Added INI-file parsing
* New flex file: btl_openib_ini.l
* New default HCA params .ini file (probably to be expanded over
time by other HCA vendors)
* Added more show_help messages for parsing problems
* Read in INI files and cache the values for later lookup
* When component opens an HCA, lookup to see if any corresponding
values were found in the INI files (ID'ed by the HCA vendor_id
and vendor_part_id)
* Added btl_openib_verbose MCA param that shows what the INI-file
stuff does (e.g., shows which MTU your HCA ends up using)
* Added btl_openib_hca_param_files as a colon-delimited list of INI
files to check for values during startup (in order,
left-to-right, just like the MCA base directory param).
* MTU is currently the only value supported in this framework.
* It is not a fatal error if we don't find params for the HCA in
the INI file(s). Instead, just print a warning. New MCA param
btl_openib_warn_no_hca_params_found can be used to disable
printing the warning.
* Add MTU to peer negotiation when making a connection
* Exchange maximum MTU; select the lesser of the two
This commit was SVN r11182.
2006-08-14 23:30:37 +04:00
|
|
|
btl_openib_eager_rdma.h \
|
|
|
|
|
btl_openib_lex.h \
|
|
|
|
|
btl_openib_lex.l \
|
|
|
|
|
btl_openib_mca.c \
|
|
|
|
|
btl_openib_mca.h \
|
|
|
|
|
btl_openib_ini.c \
|
This commit brings in two major things:
1. Galen's fine-grain control of queue pair resources in the openib
BTL.
1. Pasha's new implementation of asychronous HCA event handling.
Pasha's new implementation doesn't take much explanation, but the new
"multifrag" stuff does.
Note that "svn merge" was not used to bring this new code from the
/tmp/ib_multifrag branch -- something Bad happened in the periodic
trunk pulls on that branch making an actual merge back to the trunk
effectively impossible (i.e., lots and lots of arbitrary conflicts and
artifical changes). :-(
== Fine-grain control of queue pair resources ==
Galen's fine-grain control of queue pair resources to the OpenIB BTL
(thanks to Gleb for fixing broken code and providing additional
functionality, Pasha for finding broken code, and Jeff for doing all
the svn work and regression testing).
Prior to this commit, the OpenIB BTL created two queue pairs: one for
eager size fragments and one for max send size fragments. When the
use of the shared receive queue (SRQ) was specified (via "-mca
btl_openib_use_srq 1"), these QPs would use a shared receive queue for
receive buffers instead of the default per-peer (PP) receive queues
and buffers. One consequence of this design is that receive buffer
utilization (the size of the data received as a percentage of the
receive buffer used for the data) was quite poor for a number of
applications.
The new design allows multiple QPs to be specified at runtime. Each
QP can be setup to use PP or SRQ receive buffers as well as giving
fine-grained control over receive buffer size, number of receive
buffers to post, when to replenish the receive queue (low water mark)
and for SRQ QPs, the number of outstanding sends can also be
specified. The following is an example of the syntax to describe QPs
to the OpenIB BTL using the new MCA parameter btl_openib_receive_queues:
{{{
-mca btl_openib_receive_queues \
"P,128,16,4;S,1024,256,128,32;S,4096,256,128,32;S,65536,256,128,32"
}}}
Each QP description is delimited by ";" (semicolon) with individual
fields of the QP description delimited by "," (comma). The above
example therefore describes 4 QPs.
The first QP is:
P,128,16,4
Meaning: per-peer receive buffer QPs are indicated by a starting field
of "P"; the first QP (shown above) is therefore a per-peer based QP.
The second field indicates the size of the receive buffer in bytes
(128 bytes). The third field indicates the number of receive buffers
to allocate to the QP (16). The fourth field indicates the low
watermark for receive buffers at which time the BTL will repost
receive buffers to the QP (4).
The second QP is:
S,1024,256,128,32
Shared receive queue based QPs are indicated by a starting field of
"S"; the second QP (shown above) is therefore a shared receive queue
based QP. The second, third and fourth fields are the same as in the
per-peer based QP. The fifth field is the number of outstanding sends
that are allowed at a given time on the QP (32). This provides a
"good enough" mechanism of flow control for some regular communication
patterns.
QPs MUST be specified in ascending receive buffer size order. This
requirement may be removed prior to 1.3 release.
This commit was SVN r15474.
2007-07-18 05:15:59 +04:00
|
|
|
btl_openib_ini.h \
|
|
|
|
|
btl_openib_async.c \
|
2007-08-07 03:40:35 +04:00
|
|
|
btl_openib_async.h \
|
2007-11-28 10:18:59 +03:00
|
|
|
btl_openib_xrc.c \
|
|
|
|
|
btl_openib_xrc.h \
|
2007-08-07 03:40:35 +04:00
|
|
|
connect/base.h \
|
|
|
|
|
connect/btl_openib_connect_base.c \
|
|
|
|
|
connect/btl_openib_connect_oob.c \
|
|
|
|
|
connect/btl_openib_connect_oob.h \
|
|
|
|
|
connect/btl_openib_connect_rdma_cm.c \
|
|
|
|
|
connect/btl_openib_connect_rdma_cm.h \
|
2007-11-29 16:23:17 +03:00
|
|
|
connect/btl_openib_connect_ibcm.c \
|
|
|
|
|
connect/btl_openib_connect_ibcm.h \
|
2007-08-07 03:40:35 +04:00
|
|
|
connect/connect.h
|
Bring over all the work from the /tmp/ib-hw-detect branch. In
addition to my design and testing, it was conceptually approved by
Gil, Gleb, Pasha, Brad, and Galen. Functionally [probably somewhat
lightly] tested by Galen. We may still have to shake out some bugs
during the next few months, but it seems to be working for all the
cases that I can throw at it.
Here's a summary of the changes from that branch:
* Move MCA parameter registration to a new file (btl_openib_mca.c):
* Properly check the retun status of registering MCA params
* Check for valid values of MCA parameters
* Make help strings better
* Otherwise, the only default value of an MCA param that was
changed was max_btls; it went from 4 to -1 (meaning: use all
available)
* Properly prototyped internal functions in _component.c
* Made a bunch of functions static that didn't need to be public
* Renamed to remove "mca_" prefix from static functions
* Call new MCA param registration function
* Call new INI file read/lookup/finalize functions
* Updated a bunch of macros to be "BTL_" instead of "ORTE_"
* Be a little more consistent with return values
* Handle -1 for the max_btls MCA param
* Fixed a free() that should have been an OBJ_RELEASE()
* Some re-indenting
* Added INI-file parsing
* New flex file: btl_openib_ini.l
* New default HCA params .ini file (probably to be expanded over
time by other HCA vendors)
* Added more show_help messages for parsing problems
* Read in INI files and cache the values for later lookup
* When component opens an HCA, lookup to see if any corresponding
values were found in the INI files (ID'ed by the HCA vendor_id
and vendor_part_id)
* Added btl_openib_verbose MCA param that shows what the INI-file
stuff does (e.g., shows which MTU your HCA ends up using)
* Added btl_openib_hca_param_files as a colon-delimited list of INI
files to check for values during startup (in order,
left-to-right, just like the MCA base directory param).
* MTU is currently the only value supported in this framework.
* It is not a fatal error if we don't find params for the HCA in
the INI file(s). Instead, just print a warning. New MCA param
btl_openib_warn_no_hca_params_found can be used to disable
printing the warning.
* Add MTU to peer negotiation when making a connection
* Exchange maximum MTU; select the lesser of the two
This commit was SVN r11182.
2006-08-14 23:30:37 +04:00
|
|
|
|
2008-01-15 02:22:03 +03:00
|
|
|
if MCA_btl_openib_have_xrc
|
|
|
|
|
sources += \
|
|
|
|
|
connect/btl_openib_connect_xoob.c \
|
|
|
|
|
connect/btl_openib_connect_xoob.h
|
|
|
|
|
endif
|
|
|
|
|
|
2004-07-16 18:28:51 +04:00
|
|
|
# Make the output library in this directory, and name it either
|
|
|
|
|
# mca_<type>_<name>.la (for DSO builds) or libmca_<type>_<name>.la
|
|
|
|
|
# (for static builds).
|
|
|
|
|
|
2005-07-01 01:28:35 +04:00
|
|
|
if OMPI_BUILD_btl_openib_DSO
|
2005-05-07 01:50:30 +04:00
|
|
|
lib =
|
2005-07-01 01:28:35 +04:00
|
|
|
lib_sources =
|
|
|
|
|
component = mca_btl_openib.la
|
2005-05-07 01:50:30 +04:00
|
|
|
component_sources = $(sources)
|
2004-07-16 18:28:51 +04:00
|
|
|
else
|
2005-07-01 01:28:35 +04:00
|
|
|
lib = libmca_btl_openib.la
|
2005-05-07 01:50:30 +04:00
|
|
|
lib_sources = $(sources)
|
|
|
|
|
component =
|
|
|
|
|
component_sources =
|
2004-07-16 18:28:51 +04:00
|
|
|
endif
|
|
|
|
|
|
2007-04-12 15:19:42 +04:00
|
|
|
mcacomponentdir = $(pkglibdir)
|
2005-05-07 01:50:30 +04:00
|
|
|
mcacomponent_LTLIBRARIES = $(component)
|
2005-07-01 01:28:35 +04:00
|
|
|
mca_btl_openib_la_SOURCES = $(component_sources)
|
2005-07-27 07:38:25 +04:00
|
|
|
mca_btl_openib_la_LDFLAGS = -module -avoid-version $(btl_openib_LDFLAGS)
|
2007-12-15 16:32:02 +03:00
|
|
|
mca_btl_openib_la_LIBADD = $(btl_openib_LIBS)
|
2004-07-16 18:28:51 +04:00
|
|
|
|
2005-05-07 01:50:30 +04:00
|
|
|
noinst_LTLIBRARIES = $(lib)
|
2005-07-01 01:28:35 +04:00
|
|
|
libmca_btl_openib_la_SOURCES = $(lib_sources)
|
2005-07-27 07:38:25 +04:00
|
|
|
libmca_btl_openib_la_LDFLAGS= -module -avoid-version $(btl_openib_LDFLAGS)
|
2007-12-15 16:32:02 +03:00
|
|
|
libmca_btl_openib_la_LIBADD = $(btl_openib_LIBS)
|