Update the MX TBL and MTL section.

This commit was SVN r13879.

README

@@ -204,21 +204,37 @@ base as of this writing (28 Feb 2007):
 
   http://www.open-mpi.org/community/lists/users/2006/01/0539.php
 
-- The MX library limits the maximum message fragment size for both
+- The MX support is shared between the 2 internal devices, the MTL
-  on-node and off-node messages.  As of MX v1.0.3, the inter-node
+  and the BTL. The MTL stand for Message Transport Layer, while the
-  maximum fragment size is 32k, and the intra-node maximum fragment
+  BTL stand for Byte Transport Layer. The design of the BTL interface
-  size is 16k -- fragments sent larger than these sizes will fail.
+  in OpenMPI assumes that only naive one-sided communication
-  Open MPI automatically fragments large messages; it currently limits
+  capabilities are provided by the low level communication layers.
-  its first fragment size on MX networks to the lower of these two
+  However, modern communication layers such as MX, PSM or Portals,
-  values -- 16k.  As such, increasing the value of the MCA parameter
+  natively implement highly-optimized two-sided communication
-  named "btl_mx_first_frag_size" larger than 16k may cause failures in
+  semantics. To leverage these capabilities, OpenMPI provides the MTL
-  some cases (i.e., when using MX to send large messages to processes
+  interface to transfer messages rather than bytes.
-  on the same node); it will cause failures in all cases if it is set
+  The MTL interface implements a shorter code path and let the
-  above 32k.  Note that this only affects the *first* fragment of
+  low-level network library decides which protocol to use, depending
-  messages; latter fragments do not have this size restriction.  The
+  on message length, internal resources and other parameters
-  MCA parameter btl_mx_max_send_size can be used to vary the maximum
+  specific to the interconnect used. However, OpenMPI cannot
-  size of subsequent fragments.
+  currently use multiple MTL at once. In the case of the MX MTL,
-  *** JMS fix MX paragraph (George says it's wrong)
+  self and shared memory communications are provided by the MX
+  library. Moreover, the current MX MTL do not support message
+  pipelining resulting in lower performances in case of non
+  contiguous data-types.
+  In the case of the BTL, MCA parameters allow Open MPI to use our own
+  shared memory and self device for increased performance.
+  The BTL interface allow multiple devices to be used simultaneously.
+  For the MX BTL it is recommended that the first segment (which is
+  as a threshold between the eager and the rendez-vous protocol) should
+  always be at most 4KB, but there is no further restriction on
+  the size of subsequent fragments.
+  The MX MTL is recommended in the common case for best performance
+  on 10G hardware, when most of the data transfer cover contiguous
+  memory layouts. The MX BTL is recommended in all other cases, more
+  specifically when using multiple interconnects at the same time
+  (including TCP), transferring non contiguous data-types or when
+  using DR PML.
 
 - The OpenFabrics Enterprise Distribution (OFED) software package v1.0
   will not work properly with Open MPI v1.2 (and later) due to how its