c42ab8ea37
Commit from a long-standing Mercurial tree that ended up incorporating a lot of things: * A few fixes for CPC interface changes in all the CPCs * Attempts (but not yet finished) to fix shutdown problems in the IB CM CPC * #1319: add CTS support (i.e., initiator guarantees to send first message; automatically activated for iWARP over the RDMA CM CPC) * Some variable and function renamings to make this be generic (e.g., alloc_credit_frag became alloc_control_frag) * CPCs no longer post receive buffers; they only post a single receive buffer for the CTS if they use CTS. Instead, the main BTL now posts the main sets of receive buffers. * CPCs allocate a CTS buffer only if they're about to make a connection * RDMA CM improvements: * Use threaded mode openib fd monitoring to wait for for RDMA CM events * Synchronize endpoint finalization and disconnection between main thread and service thread to avoid/fix some race conditions * Converted several structs to be OBJs so that we can use reference counting to know when to invoke destructors * Make some new OBJ's have opal_list_item_t's as their base, thereby eliminating the need for the local list_item_t type * Renamed many variables to be internally consistent * Centralize the decision in an inline function as to whether this process or the remote process is supposed to be the initiator * Add oodles of OPAL_OUTPUT statements for debugging (hard-wired to output stream -1; to be activated by developers if they want/need them) * Use rdma_create_qp() instead of ibv_create_qp() * openib fd monitoring improvements: * Renamed a bunch of functions and variables to be a little more obvious as to their true function * Use pipes to communicate between main thread and service thread * Add ability for main thread to invoke a function back on the service thread * Ensure to set initiator_depth and responder_resources properly, but putting max_qp_rd_ataom and ma_qp_init_rd_atom in the modex (see rdma_connect(3)) * Ensure to set the source IP address in rdma_resolve() to ensure that we select the correct OpenFabrics source port * Make new MCA param: openib_btl_connect_rdmacm_resolve_timeout * Other improvements: * btl_openib_device_type MCA param: can be "iw" or "ib" or "all" (or "infiniband" or "iwarp") * Somewhat improved error handling * Bunches of spelling fixes in comments, VERBOSE, and OUTPUT statements * Oodles of little coding style fixes * Changed shutdown ordering of btl; the device is now an OBJ with ref counting for destruction * Added some more show_help error messages * Change configury to only build IBCM / RDMACM if we have threads (because we need a progress thread) This commit was SVN r19686. The following Trac tickets were found above: Ticket 1210 --> https://svn.open-mpi.org/trac/ompi/ticket/1210
74 строки
1.7 KiB
C
74 строки
1.7 KiB
C
/*
|
|
* Copyright (c) 2008 Cisco, Inc. All rights reserved.
|
|
*
|
|
* $COPYRIGHT$
|
|
*
|
|
* Additional copyrights may follow
|
|
*
|
|
* $HEADER$
|
|
*/
|
|
|
|
#ifndef OMPI_BTL_OPENIB_FD_H_
|
|
#define OMPI_BTL_OPENIB_FD_H_
|
|
|
|
#include "ompi_config.h"
|
|
|
|
BEGIN_C_DECLS
|
|
|
|
/**
|
|
* Typedef for fd callback function
|
|
*/
|
|
typedef void *(ompi_btl_openib_fd_event_callback_fn_t)(int fd, int flags,
|
|
void *context);
|
|
|
|
/**
|
|
* Typedef for generic callback function
|
|
*/
|
|
typedef void *(ompi_btl_openib_fd_main_callback_fn_t)(void *context);
|
|
|
|
/**
|
|
* Initialize fd monitoring.
|
|
* Called by the main thread.
|
|
*/
|
|
int ompi_btl_openib_fd_init(void);
|
|
|
|
/**
|
|
* Start monitoring an fd.
|
|
* Called by main or service thread; callback will be in service thread.
|
|
*/
|
|
int ompi_btl_openib_fd_monitor(int fd, int flags,
|
|
ompi_btl_openib_fd_event_callback_fn_t *callback,
|
|
void *context);
|
|
|
|
/**
|
|
* Stop monitoring an fd.
|
|
* Called by main or service thread; callback will be in service thread.
|
|
*/
|
|
int ompi_btl_openib_fd_unmonitor(int fd,
|
|
ompi_btl_openib_fd_event_callback_fn_t *callback,
|
|
void *context);
|
|
|
|
/**
|
|
* Run a function in the service thread.
|
|
* Called by the main thread.
|
|
*/
|
|
int ompi_btl_openib_fd_run_in_service(ompi_btl_openib_fd_main_callback_fn_t callback,
|
|
void *context);
|
|
|
|
/**
|
|
* Run a function in the main thread.
|
|
* Called by the service thread.
|
|
*/
|
|
int ompi_btl_openib_fd_run_in_main(ompi_btl_openib_fd_main_callback_fn_t callback,
|
|
void *context);
|
|
|
|
/**
|
|
* Finalize fd monitoring.
|
|
* Called by the main thread.
|
|
*/
|
|
int ompi_btl_openib_fd_finalize(void);
|
|
|
|
END_C_DECLS
|
|
|
|
#endif
|