ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
openmpi/ompi/mca/btl/usnic/btl_usnic_send.c

184 строки
6.1 KiB
C
Исходник Обычный вид История

First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
/*
* Copyright (c) 2004-2008 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* Copyright (c) 2004-2011 The University of Tennessee and The University
* of Tennessee Research Foundation. All rights
* reserved.
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
* University of Stuttgart. All rights reserved.
* Copyright (c) 2004-2005 The Regents of the University of California.
* All rights reserved.
* Copyright (c) 2006 Sandia National Laboratories. All rights
* reserved.
* Copyright (c) 2008-2013 Cisco Systems, Inc. All rights reserved.
* Copyright (c) 2012 Los Alamos National Security, LLC. All rights
* reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
*
* $HEADER$
*/
#include "ompi_config.h"
#include <infiniband/verbs.h>
#include "opal_stdint.h"
#include "ompi/constants.h"
#include "ompi/mca/btl/btl.h"
#include "ompi/mca/btl/base/base.h"
#include "ompi/mca/common/verbs/common_verbs.h"
#include "btl_usnic.h"
#include "btl_usnic_frag.h"
#include "btl_usnic_util.h"
#include "btl_usnic_send.h"
#include "btl_usnic_ack.h"
/*
* This function is called when a send of a full-fragment segment completes
* Return the WQE and also return the segment if no ACK pending
*/
void
ompi_btl_usnic_frag_send_complete(ompi_btl_usnic_module_t *module,
ompi_btl_usnic_send_segment_t *sseg)
{
ompi_btl_usnic_send_frag_t *frag;
frag = sseg->ss_parent_frag;
/* Reap a frag that was sent */
--sseg->ss_send_posted;
--frag->sf_seg_post_cnt;
/* checks for returnability made inside */
ompi_btl_usnic_send_frag_return_cond(module, frag);
/* do bookkeeping */
++frag->sf_endpoint->endpoint_send_credits;
++module->mod_channels[sseg->ss_channel].sd_wqe;
/* see if this endpoint needs to be made ready-to-send */
ompi_btl_usnic_check_rts(frag->sf_endpoint);
}
/*
* This function is called when a send segment completes
* Return the WQE and also return the segment if no ACK pending
*/
void
ompi_btl_usnic_chunk_send_complete(ompi_btl_usnic_module_t *module,
ompi_btl_usnic_send_segment_t *sseg)
{
ompi_btl_usnic_send_frag_t *frag;
frag = sseg->ss_parent_frag;
/* Reap a frag that was sent */
--sseg->ss_send_posted;
--frag->sf_seg_post_cnt;
if (sseg->ss_send_posted == 0 && !sseg->ss_ack_pending) {
Cisco CSCuj22803: fixes for Bsend changes required to support MPI_Bsend(). Introduces concept of attaching a buffer to a large segment that the PML can scribble into and we will send from. The reason we don't use a pinned buffer and send directly from that is that usnic_verbs does not (yes) support num_sge>1 for regular sends. This means the data gets copied twice, but that is unavoidable. changed the logic in handle_large_send to be more sensible Incorporated David's review comments This commit was SVN r29184.
2013-09-17 07:27:39 +00:00
ompi_btl_usnic_release_send_segment(module, frag, sseg);
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
}
/* done with whole fragment? */
/* checks for returnability made inside */
ompi_btl_usnic_send_frag_return_cond(module, frag);
/* do bookkeeping */
++frag->sf_endpoint->endpoint_send_credits;
++module->mod_channels[sseg->ss_channel].sd_wqe;
/* see if this endpoint needs to be made ready-to-send */
ompi_btl_usnic_check_rts(frag->sf_endpoint);
}
usnic: pack via convertor on the fly If we need to use a convertor, go back to stashing that convertor in the frag and populating segments "on the fly" (in ompi_btl_usnic_module_progress_sends). Previously we would pack into a chain of chunk segments at prepare_src time, unnecessarily consuming additional memory. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> Reviewed-by: Reese Faucette <rfaucett@cisco.com> This commit was SVN r29592.
2013-11-04 22:52:03 +00:00
/* Responsible for completing non-fastpath parts of a put or send operation,
* including initializing any large frag bookkeeping fields and enqueuing the
* frag on the endpoint.
*
* This routine lives in this file to help prevent automatic inlining by the
* compiler.
*
* The "tag" only applies to sends. */
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
int
usnic: pack via convertor on the fly If we need to use a convertor, go back to stashing that convertor in the frag and populating segments "on the fly" (in ompi_btl_usnic_module_progress_sends). Previously we would pack into a chain of chunk segments at prepare_src time, unnecessarily consuming additional memory. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> Reviewed-by: Reese Faucette <rfaucett@cisco.com> This commit was SVN r29592.
2013-11-04 22:52:03 +00:00
ompi_btl_usnic_finish_put_or_send(
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
ompi_btl_usnic_module_t *module,
ompi_btl_usnic_endpoint_t *endpoint,
ompi_btl_usnic_send_frag_t *frag,
mca_btl_base_tag_t tag)
{
int rc;
ompi_btl_usnic_small_send_frag_t *sfrag;
ompi_btl_usnic_send_segment_t *sseg;
/*
* If this is small, need to do the copyin now.
* We don't do this earlier in case we got lucky and were
* able to do an inline send. We did not, so here we are...
*/
if (frag->sf_base.uf_type == OMPI_BTL_USNIC_FRAG_SMALL_SEND) {
sfrag = (ompi_btl_usnic_small_send_frag_t *)frag;
sseg = &sfrag->ssf_segment;
usnic: pack via convertor on the fly If we need to use a convertor, go back to stashing that convertor in the frag and populating segments "on the fly" (in ompi_btl_usnic_module_progress_sends). Previously we would pack into a chain of chunk segments at prepare_src time, unnecessarily consuming additional memory. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> Reviewed-by: Reese Faucette <rfaucett@cisco.com> This commit was SVN r29592.
2013-11-04 22:52:03 +00:00
/* Copy in user data if there is any, collapsing 2 segments into 1.
* We already packed via the convertor if necessary, so we only need to
* handle the simple memcpy case here.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
*/
Per RFC: Remove des_src and des_dst members from the mca_btl_base_segment_t and replace them with des_local and des_remote This change also updates the BTL version to 3.0.0. This commit does not represent the final version of BTL 3.0.0. More changes are coming. In making this change I updated all of the BTLs as well as BTL user's to use the new structure members. Please evaluate your component to ensure the changes are correct. RFC text: This is the first of several BTL interface changes I am proposing for the 1.9/2.0 release series. What: Change naming of btl descriptor members. I propose we change des_src and des_dst (and their associated counts) to be des_local and des_remote. For receive callbacks the des_local member will be used to communicate the segment information to the callback. The proposed change will include updating all of the doxygen in btl.h as well as updating all BTLs and BTL users to use the new naming scheme. Why: My btl usage makes use of both put and get operations on the same descriptor. With the current naming scheme I need to ensure that there is consistency beteen the segments described in des_src and des_dst depending on whether a put or get operation is executed. Additionally, the current naming prevents BTLs that do not require prepare/RMA matched operations (do not set MCA_BTL_FLAGS_RDMA_MATCHED) from executing multiple simultaneous put AND get operations. At the moment the descriptor can only be used with one or the other. The naming change makes it easier for BTL users to setup/modify descriptors for RMA operations as the local segment and remote segment are always in the same member field. The only issue I forsee with this change is that it will require a little more work to move BTL fixes to the 1.8 release series. This commit was SVN r32196.
2014-07-10 16:31:15 +00:00
if (frag->sf_base.uf_base.des_local_count > 1) {
usnic: pack via convertor on the fly If we need to use a convertor, go back to stashing that convertor in the frag and populating segments "on the fly" (in ompi_btl_usnic_module_progress_sends). Previously we would pack into a chain of chunk segments at prepare_src time, unnecessarily consuming additional memory. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> Reviewed-by: Reese Faucette <rfaucett@cisco.com> This commit was SVN r29592.
2013-11-04 22:52:03 +00:00
/* no convertor */
usnic: Fix compile errors related to r32196 This commit was SVN r32198. The following SVN revision numbers were found above: r32196 --> open-mpi/ompi@a14e0f10d4a083c66652f92a5300aef089058a08
2014-07-10 17:18:03 +00:00
assert(NULL != frag->sf_base.uf_local_seg[1].seg_addr.pval);
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
usnic: Fix compile errors related to r32196 This commit was SVN r32198. The following SVN revision numbers were found above: r32196 --> open-mpi/ompi@a14e0f10d4a083c66652f92a5300aef089058a08
2014-07-10 17:18:03 +00:00
memcpy(((char *)(intptr_t)frag->sf_base.uf_local_seg[0].seg_addr.lval +
frag->sf_base.uf_local_seg[0].seg_len),
frag->sf_base.uf_local_seg[1].seg_addr.pval,
frag->sf_base.uf_local_seg[1].seg_len);
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
/* update 1st segment length */
Per RFC: Remove des_src and des_dst members from the mca_btl_base_segment_t and replace them with des_local and des_remote This change also updates the BTL version to 3.0.0. This commit does not represent the final version of BTL 3.0.0. More changes are coming. In making this change I updated all of the BTLs as well as BTL user's to use the new structure members. Please evaluate your component to ensure the changes are correct. RFC text: This is the first of several BTL interface changes I am proposing for the 1.9/2.0 release series. What: Change naming of btl descriptor members. I propose we change des_src and des_dst (and their associated counts) to be des_local and des_remote. For receive callbacks the des_local member will be used to communicate the segment information to the callback. The proposed change will include updating all of the doxygen in btl.h as well as updating all BTLs and BTL users to use the new naming scheme. Why: My btl usage makes use of both put and get operations on the same descriptor. With the current naming scheme I need to ensure that there is consistency beteen the segments described in des_src and des_dst depending on whether a put or get operation is executed. Additionally, the current naming prevents BTLs that do not require prepare/RMA matched operations (do not set MCA_BTL_FLAGS_RDMA_MATCHED) from executing multiple simultaneous put AND get operations. At the moment the descriptor can only be used with one or the other. The naming change makes it easier for BTL users to setup/modify descriptors for RMA operations as the local segment and remote segment are always in the same member field. The only issue I forsee with this change is that it will require a little more work to move BTL fixes to the 1.8 release series. This commit was SVN r32196.
2014-07-10 16:31:15 +00:00
frag->sf_base.uf_base.des_local_count = 1;
usnic: Fix compile errors related to r32196 This commit was SVN r32198. The following SVN revision numbers were found above: r32196 --> open-mpi/ompi@a14e0f10d4a083c66652f92a5300aef089058a08
2014-07-10 17:18:03 +00:00
frag->sf_base.uf_local_seg[0].seg_len +=
frag->sf_base.uf_local_seg[1].seg_len;
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
}
sseg->ss_base.us_sg_entry[0].length =
sizeof(ompi_btl_usnic_btl_header_t) + frag->sf_size;
/* use standard channel */
sseg->ss_channel = USNIC_DATA_CHANNEL;
Cisco CSCuj12520: various problems running c_fence_put_1 - tag needs to be sent in *our* header, not the PML header - usnic_alloc() should return smaller value if too much data requested - be careful about callbacks vs removing items from lists (we need to remove from outr lists *before* the callback) - improve send callback handling - add some more MSGDEBUG2 logging and cleanup This commit was SVN r29181.
2013-09-17 07:20:44 +00:00
sseg->ss_base.us_btl_header->tag = tag;
usnic: Use INLINE flag more often, saving the DMA is useful. Authored-by: Reese Faucette <rfaucett@cisco.com> Reviewed-by: Jeff Squyres <jsquyres@cisco.com> cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30833. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-26 07:39:53 +00:00
usnic: Fix compile errors related to r32196 This commit was SVN r32198. The following SVN revision numbers were found above: r32196 --> open-mpi/ompi@a14e0f10d4a083c66652f92a5300aef089058a08
2014-07-10 17:18:03 +00:00
if (frag->sf_base.uf_local_seg[0].seg_len < module->tiny_mtu) {
usnic: Use INLINE flag more often, saving the DMA is useful. Authored-by: Reese Faucette <rfaucett@cisco.com> Reviewed-by: Jeff Squyres <jsquyres@cisco.com> cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30833. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-26 07:39:53 +00:00
sseg->ss_send_desc.send_flags |= IBV_SEND_INLINE;
}
Cisco CSCuj12520: various problems running c_fence_put_1 - tag needs to be sent in *our* header, not the PML header - usnic_alloc() should return smaller value if too much data requested - be careful about callbacks vs removing items from lists (we need to remove from outr lists *before* the callback) - improve send callback handling - add some more MSGDEBUG2 logging and cleanup This commit was SVN r29181.
2013-09-17 07:20:44 +00:00
} else {
ompi_btl_usnic_large_send_frag_t *lfrag;
usnic: pack via convertor on the fly If we need to use a convertor, go back to stashing that convertor in the frag and populating segments "on the fly" (in ompi_btl_usnic_module_progress_sends). Previously we would pack into a chain of chunk segments at prepare_src time, unnecessarily consuming additional memory. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> Reviewed-by: Reese Faucette <rfaucett@cisco.com> This commit was SVN r29592.
2013-11-04 22:52:03 +00:00
/* Save info about the frag so that future invocations of
* usnic_handle_large_send can generate segments to put on the wire. */
Cisco CSCuj12520: various problems running c_fence_put_1 - tag needs to be sent in *our* header, not the PML header - usnic_alloc() should return smaller value if too much data requested - be careful about callbacks vs removing items from lists (we need to remove from outr lists *before* the callback) - improve send callback handling - add some more MSGDEBUG2 logging and cleanup This commit was SVN r29181.
2013-09-17 07:20:44 +00:00
lfrag = (ompi_btl_usnic_large_send_frag_t *)frag;
lfrag->lsf_tag = tag;
Cisco CSCuj22803: fixes for Bsend changes required to support MPI_Bsend(). Introduces concept of attaching a buffer to a large segment that the PML can scribble into and we will send from. The reason we don't use a pinned buffer and send directly from that is that usnic_verbs does not (yes) support num_sge>1 for regular sends. This means the data gets copied twice, but that is unavoidable. changed the logic in handle_large_send to be more sensible Incorporated David's review comments This commit was SVN r29184.
2013-09-17 07:27:39 +00:00
lfrag->lsf_cur_offset = 0;
usnic: pack via convertor on the fly If we need to use a convertor, go back to stashing that convertor in the frag and populating segments "on the fly" (in ompi_btl_usnic_module_progress_sends). Previously we would pack into a chain of chunk segments at prepare_src time, unnecessarily consuming additional memory. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> Reviewed-by: Reese Faucette <rfaucett@cisco.com> This commit was SVN r29592.
2013-11-04 22:52:03 +00:00
lfrag->lsf_cur_ptr = lfrag->lsf_des_src[0].seg_addr.pval;
Cisco CSCuj22803: fixes for Bsend changes required to support MPI_Bsend(). Introduces concept of attaching a buffer to a large segment that the PML can scribble into and we will send from. The reason we don't use a pinned buffer and send directly from that is that usnic_verbs does not (yes) support num_sge>1 for regular sends. This means the data gets copied twice, but that is unavoidable. changed the logic in handle_large_send to be more sensible Incorporated David's review comments This commit was SVN r29184.
2013-09-17 07:27:39 +00:00
lfrag->lsf_cur_sge = 0;
usnic: pack via convertor on the fly If we need to use a convertor, go back to stashing that convertor in the frag and populating segments "on the fly" (in ompi_btl_usnic_module_progress_sends). Previously we would pack into a chain of chunk segments at prepare_src time, unnecessarily consuming additional memory. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> Reviewed-by: Reese Faucette <rfaucett@cisco.com> This commit was SVN r29592.
2013-11-04 22:52:03 +00:00
lfrag->lsf_bytes_left_in_sge = lfrag->lsf_des_src[0].seg_len;
lfrag->lsf_bytes_left = frag->sf_size;
if (lfrag->lsf_pack_on_the_fly) {
lfrag->lsf_pack_bytes_left = frag->sf_size;
} else {
/* we pre-packed the convertor into a chain in prepare_src */
lfrag->lsf_pack_bytes_left = 0;
Cisco CSCuj22803: fixes for Bsend changes required to support MPI_Bsend(). Introduces concept of attaching a buffer to a large segment that the PML can scribble into and we will send from. The reason we don't use a pinned buffer and send directly from that is that usnic_verbs does not (yes) support num_sge>1 for regular sends. This means the data gets copied twice, but that is unavoidable. changed the logic in handle_large_send to be more sensible Incorporated David's review comments This commit was SVN r29184.
2013-09-17 07:27:39 +00:00
}
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-19 22:13:58 +00:00
}
/* queue this fragment into the send engine */
rc = ompi_btl_usnic_endpoint_enqueue_frag(endpoint, frag);
return rc;
}
Копировать постоянную ссылку