openmpi/opal/mca/common/libfabric/libfabric/man/fi_av.3

.TH fi_av 3 "2015\-02\-01" "Libfabric Programmer\[aq]s Manual" "Libfabric v1.0.0-rc1"
.SH NAME
.PP
fi_av - Address vector operations
.PP
fi_av_open / fi_close : Open or close an address vector
.PP
fi_av_bind : Associate an address vector with an event queue.
.PP
fi_av_insert / fi_av_insertsvc / fi_av_remove : Insert/remove an address
into/from the address vector.
.PP
fi_av_lookup : Retrieve an address stored in the address vector.
.PP
fi_av_straddr : Convert an address into a printable string.
.SH SYNOPSIS
.IP
.nf
\f[C]
#include\ <rdma/fi_domain.h>

int\ fi_av_open(struct\ fid_domain\ *domain,\ struct\ fi_av_attr\ *attr,
\ \ \ \ struct\ fid_av\ **av,\ void\ *context);

int\ fi_close(struct\ fid\ *av);

int\ fi_av_bind(struct\ fid_av\ *av,\ struct\ fid\ *eq,\ uint64_t\ flags);

int\ fi_av_insert(struct\ fid_av\ *av,\ void\ *addr,\ size_t\ count,
\ \ \ \ fi_addr_t\ *fi_addr,\ uint64_t\ flags,\ void\ *context);

int\ fi_av_insertsvc(struct\ fid_av\ *av,\ const\ char\ *node,
\ \ \ \ const\ char\ *service,\ fi_addr_t\ *fi_addr,\ uint64_t\ flags,
\ \ \ \ void\ *context);

int\ fi_av_insertsym(struct\ fid_av\ *av,\ const\ char\ *node,
\ \ \ \ size_t\ nodecnt,\ const\ char\ *service,\ size_t\ svccnt,
\ \ \ \ fi_addr_t\ *fi_addr,\ uint64_t\ flags,\ void\ *context);

int\ fi_av_remove(struct\ fid_av\ *av,\ fi_addr_t\ *fi_addr,\ size_t\ count,
\ \ \ \ uint64_t\ flags);

int\ fi_av_lookup(struct\ fid_av\ *av,\ fi_addr_t\ fi_addr,
\ \ \ \ void\ *addr,\ size_t\ *addrlen);

fi_addr_t\ fi_rx_addr(fi_addr_t\ fi_addr,\ int\ rx_index,
\ \ \ \ \ \ int\ rx_ctx_bits);

const\ char\ *\ fi_av_straddr(struct\ fid_av\ *av,\ const\ void\ *addr,
\ \ \ \ \ \ void\ *buf,\ size_t\ len);
\f[]
.fi
.SH ARGUMENTS
.PP
\f[I]domain\f[] : Resource domain
.PP
\f[I]av\f[] : Address vector
.PP
\f[I]eq\f[] : Event queue
.PP
\f[I]attr\f[] : Address vector attributes
.PP
\f[I]context\f[] : User specified context associated with the address
vector or insert operation.
.PP
\f[I]addr\f[] : Buffer containing one or more addresses to insert into
address vector.
.PP
\f[I]addrlen\f[] : On input, specifies size of addr buffer.
On output, stores number of bytes written to addr buffer.
.PP
\f[I]fi_addr\f[] : For insert, a reference to an array where returned
fabric addresses will be written.
For remove, one or more fabric addresses to remove.
.PP
\f[I]count\f[] : Number of addresses to insert/remove from an AV.
.PP
\f[I]flags\f[] : Additional flags to apply to the operation.
.SH DESCRIPTION
.PP
Address vectors are used to map higher level addresses, which may be
more natural for an application to use, into fabric specific addresses.
The mapping of addresses is fabric and provider specific, but may
involve lengthy address resolution and fabric management protocols.
AV operations are synchronous by default, but may be set to operate
asynchronously by specifying the FI_EVENT flag to \f[C]fi_av_open\f[].
When requesting asynchronous operation, the application must first bind
an event queue to the AV before inserting addresses.
.SS fi_av_open
.PP
fi_av_open allocates or opens an address vector.
The properties and behavior of the address vector are defined by
\f[C]struct\ fi_av_attr\f[].
.IP
.nf
\f[C]
struct\ fi_av_attr\ {
\ \ \ \ enum\ fi_av_type\ \ type;\ \ \ \ \ \ \ \ /*\ type\ of\ AV\ */
\ \ \ \ int\ \ \ \ \ \ \ \ \ \ \ \ \ \ rx_ctx_bits;\ /*\ address\ bits\ to\ identify\ rx\ ctx\ */
\ \ \ \ size_t\ \ \ \ \ \ \ \ \ \ \ count;\ \ \ \ \ \ \ /*\ #\ entries\ for\ AV\ */
\ \ \ \ size_t\ \ \ \ \ \ \ \ \ \ \ ep_per_node;\ /*\ #\ endpoints\ per\ fabric\ address\ */
\ \ \ \ const\ char\ \ \ \ \ \ \ *name;\ \ \ \ \ \ \ /*\ system\ name\ of\ AV\ */
\ \ \ \ void\ \ \ \ \ \ \ \ \ \ \ \ \ *map_addr;\ \ \ /*\ base\ mmap\ address\ */
\ \ \ \ uint64_t\ \ \ \ \ \ \ \ \ flags;\ \ \ \ \ \ \ /*\ operation\ flags\ */
};
\f[]
.fi
.PP
\f[I]type\f[] : An AV type corresponds to a conceptual implementation of
an address vector.
The type specifies how an application views data stored in the AV,
including how it may be accessed.
Valid values are:
.IP \[bu] 2
\f[I]FI_AV_MAP\f[] : Addresses which are inserted into an AV are mapped
to a native fabric address for use by the application.
The use of FI_AV_MAP requires that an application store the returned
fi_addr_t value that is associated with each inserted address.
The advantage of using FI_AV_MAP is that the returned fi_addr_t value
may contain encoded address data, which is immediately available when
processing data transfer requests.
This can eliminate or reduce the number of memory lookups needed when
initiating a transfer.
The disadvantage of FI_AV_MAP is the increase in memory usage needed to
store the returned addresses.
Addresses are stored in the AV using a provider specific mechanism,
including, but not limited to a tree, hash table, or maintained on the
heap.
.IP \[bu] 2
\f[I]FI_AV_TABLE\f[] : Addresses which are inserted into an AV of type
FI_AV_TABLE are accessible using a simple index.
Conceptually, the AV may be treated as an array of addresses, though the
provider may implement the AV using a variety of mechanisms.
When FI_AV_TABLE is used, the returned fi_addr_t is an index, with the
index for an inserted address the same as its insertion order into the
table.
The index of the first address inserted into an FI_AV_TABLE will be 0,
and successive insertions will be given sequential indices.
Sequential indices will be assigned across insertion calls on the same
AV.
.PP
\f[I]Receive Context Bits (rx_ctx_bits)\f[] : The receive context bits
field is only for use with scalable endpoints.
It indicates the number of bits reserved in a returned fi_addr_t, which
will be used to identify a specific target receive context.
See fi_rx_addr() and fi_endpoint(3) for additional details on receive
contexts.
The requested number of bits should be selected such that 2 ^
rx_ctx_bits >= rx_ctx_cnt for the endpoint.
.PP
\f[I]count\f[] : Indicates the expected number of addresses that will be
inserted into the AV.
The provider uses this to optimize resource allocations.
.PP
\f[I]ep_per_node\f[] : This field indicates the number of endpoints that
will be associated with a specific fabric, or network, address.
If the number of endpoints per node is unknown, this value should be set
to 0.
The provider uses this value to optimize resource allocations.
For example, distributed, parallel applications may set this to the
number of processes allocated per node, times the number of endpoints
each process will open.
.PP
\f[I]name\f[] : An optional system name associated with the address
vector to create or open.
Address vectors may be shared across multiple processes which access the
same named domain on the same node.
The name field allows the underlying provider to identify a shared AV.
.PP
If the name field is non-NULL and the AV is not opened for read-only
access, a named AV will be created, if it does not already exist.
.PP
\f[I]map_addr\f[] : The map_addr determines the base FI_ADDR address
that a provider should use when sharing an AV of type FI_AV_MAP between
processes.
Processes that provide the same value for map_addr to a shared AV may
use the same FI_ADDR values returned from an fi_av_insert call.
.PP
The map_addr may be used by the provider to mmap memory allocated for a
shared AV between processes; however, the provider is not required to
use the map_addr in this fashion.
The only requirement is that an FI_ADDR returned as part of an
fi_av_insert call on one process is usable on another process which
opens an AV of the same name at the same map_addr value.
The relationship between the map_addr and any returned FI_ADDR is not
defined.
.PP
If name is non-NULL and map_addr is 0, then the map_addr used by the
provider will be returned through the attribute structure.
The map_addr field is ignored if name is NULL.
.PP
\f[I]flags\f[] : The following flags may be used when opening an AV.
.IP \[bu] 2
\f[I]FI_EVENT\f[] : When the flag FI_EVENT is specified, all insert
operations on this AV will occur asynchronously.
There will be one EQ error entry generated for each failed address
insertion, followed by one non-error event indicating that the insertion
operation has completed.
There will always be one non-error completion event for each insert
operation, even if all addresses fail.
The context field in all completions will be the context specified to
the insert call, and the data field in the final completion entry will
report the number of addresses successfully inserted.
: \  : If an AV is opened with FI_EVENT, any insertions attempted before
an EQ is bound to the AV will fail with -FI_ENOEQ.
: \  : Error completions for failed insertions will contain the index of
the failed address in the index field of the error completion entry.
: \  : Note that the order of delivery of insert completions may not
match the order in which the calls to fi_av_insert were made.
The only guarantee is that all error completions for a given call to
fi_av_insert will precede the single associated non-error completion.
.IP \[bu] 2
\f[I]FI_READ\f[] : Opens an AV for read-only access.
An AV opened for read-only access must be named (name attribute
specified), and the AV must exist.
.IP \[bu] 2
\f[I]FI_SYMMETRIC\f[] : Indicates that each node will be associated with
the same number of endpoints, the same transport addresses will be
allocated on each node, and the transport addresses will be sequential.
This feature targets distributed applications on large fabrics and
allows for highly-optimized storage of remote endpoint addressing.
.SS fi_close
.PP
The fi_close call is used to release all resources associated with an
address vector.
Note that any events queued on an event queue referencing the AV are
left untouched.
It is recommended that callers retrieve all events associated with the
AV before closing it.
.PP
When closing the address vector, there must be no opened endpoints
associated with the AV.
If resources are still associated with the AV when attempting to close,
the call will return -FI_EBUSY.
.SS fi_av_bind
.PP
Associates an event queue with the AV.
If an AV has been opened with \f[C]FI_EVENT\f[], then an event queue
must be bound to the AV before any insertion calls are attempted.
Any calls to insert addresses before an event queue has been bound will
fail with \f[C]-FI_ENOEQ\f[].
.SS fi_av_insert
.PP
The fi_av_insert call inserts zero or more addresses into an AV.
The number of addresses is specified through the count parameter.
The addr parameter references an array of addresses to insert into the
AV.
Addresses inserted into an address vector must be in the same format as
specified in struct fi_info:addr_format for the corresponding domain.
A NULL value for an address may be used to indicate that an entry should
be associated with \[aq]any\[aq] address (similar to the IPv4 address of
0.0.0.0).
.PP
For AV\[aq]s of type FI_AV_MAP, once inserted addresses have been
mapped, the mapped values are written into the buffer referenced by
fi_addr.
The fi_addr buffer must remain valid until the AV insertion has
completed and an event has been generated to an associated event queue.
The value of the returned fi_addr should be considered opaque by the
application for AVs of type FI_AV_MAP.
The returned value may point to an internal structure or a provider
specific encoding of low-level addressing data, for example.
In the latter case, use of FI_AV_MAP may be able to avoid memory
references during data transfer operations.
.PP
For AV\[aq]s of type FI_AV_TABLE, addresses are placed into the table in
order.
That is, the first address inserted may be referenced at index 0.
The fi_addr parameter may be NULL in this case.
Otherwise, fi_addr must reference an array of fi_addr_t, and the buffer
must remain valid until the insertion operation completes.
Note that if fi_addr is NULL and synchronous operation is requested,
individual insertion failures cannot be reported and the application
must use other calls, such as \f[C]fi_av_lookup\f[] to learn which
specific addresses failed to insert.
When addresses are inserted into an AV of type FI_AV_TABLE, the assigned
fi_addr values will be simple indices corresponding to the entry into
the table where the address was inserted.
Addresses are indexed in order of their insertion.
Index values accumulate across successive insert calls in the order the
calls are made, not necessarily in the order the insertions complete.
.PP
\f[I]flags\f[] : The following flag may be passed to fi_av_insert
.IP \[bu] 2
\f[I]FI_MORE\f[] : In order to allow optimized address insertion, the
application may specify the FI_MORE flag to the insert call to give a
hint to the provider that more insertion requests will follow, allowing
the provider to aggregate insertion requests if desired.
Providers are free to ignore FI_MORE.
.SS fi_av_insertsvc
.PP
The fi_av_insertsvc call behaves similar to fi_av_insert, but allows the
application to specify the node and service names, similar to the
fi_getinfo inputs, rather than an encoded address.
The node and service parameters are defined the same as fi_getinfo(3).
Node should be a string that corresponds to a hostname or network
address.
The service string corresponds to a textual representation of a
transport address.
.SS fi_av_insertsym
.PP
fi_av_insertsym performs a symmetric insert that inserts a sequential
range of nodes and/or service addresses into an AV.
The svccnt parameter indicates the number of transport (endpoint)
addresses to insert into the AV for each node address, with the service
parameter specifying the starting transport address.
Inserted transport addresses will be of the range {service, service +
svccnt - 1}, inclusive.
All service addresses for a node will be inserted before the next node
is inserted.
.PP
The nodecnt parameter indicates the number of node (network) addresses
to insert into the AV, with the node parameter specifying the starting
node address.
Inserted node addresses will be of the range {node, node + nodecnt - 1},
inclusive.
If node is a non-numeric string, such as a hostname, it must contain a
numeric suffix if nodecnt > 1.
.PP
As an example, if node = "10.1.1.1", nodecnt = 2, service = "5000", and
svccnt = 2, the following addresses will be inserted into the AV in the
order shown: 10.1.1.1:5000, 10.1.1.1:5001, 10.1.1.2:5000, 10.1.1.1:5001.
If node were replaced by the hostname "host10", the addresses would be:
host10:5000, host10:5001, host11:5000, host11:5001.
.PP
The total number of inserted addresses will be nodecnt x svccnt.
.SS fi_av_remove
.PP
fi_av_remove removes a set of addresses from an address vector.
All resources associated with the indicated addresses are released, and
no future references to either the mapped address (in the case of
FI_AV_MAP) or index (FI_AV_TABLE) are allowed.
.PP
The use of fi_av_remove is an optimization that applications may use to
free memory allocated with addresses that will no longer be accessed.
Inserted addresses are not required to be removed.
fi_av_close will automatically cleanup any resources associated with
addresses remaining in the AV when it is invoked.
.SS fi_av_lookup
.PP
This call returns the address stored in the address vector that
corresponds to the given fi_addr.
The returned address is the same format as those stored by the AV.
On input, the addrlen parameter should indicate the size of the addr
buffer.
If the actual address is larger than what can fit into the buffer, it
will be truncated.
On output, addrlen is set to the size of the buffer needed to store the
address, which may be larger than the input value.
.SS fi_rx_addr
.PP
This function is used to convert an endpoint address, returned by
fi_av_insert, into an address that specifies a target receive context.
The specified fi_addr parameter must either be a value returned from
fi_av_insert, in the case of FI_AV_MAP, or an index, in the case of
FI_AV_TABLE.
The value for rx_ctx_bits must match that specified in the AV attributes
for the given address.
.PP
Connected endpoints that support multiple receive contexts, but are not
associated with address vectors should specify FI_ADDR_NOTAVAIL for the
fi_addr parameter.
.SS fi_av_straddr
.PP
The fi_av_straddr function converts the provided address into a
printable string.
The specified address must be of the same format as those stored by the
AV, though the address itself is not required to have been inserted.
On input, the len parameter should specify the size of the buffer
referenced by buf.
On output, the actual size needed to write the entire string will be
returned.
This size may be larger than the input len.
If the provided buffer is too small, the results will be truncated.
fi_av_straddr returns a pointer to buf.
.SH NOTES
.PP
Providers may implement AV\[aq]s using a variety of mechanisms.
Specifically, a provider may begin resolving inserted addresses as soon
as they have been added to an AV, even if asynchronous operation has
been specified.
Similarly, a provider may lazily release resources from removed entries.
.SH RETURN VALUES
.PP
The insert calls return the number of addresses successfully inserted or
the number of asynchronous insertions initiated if FI_EVENT is set.
.PP
Other calls return 0 on success.
.PP
On error, a negative value corresponding to fabric errno is returned.
Fabric errno values are defined in \f[C]rdma/fi_errno.h\f[].
.SH ERRORS
.SH SEE ALSO
.PP
\f[C]fi_getinfo\f[](3), \f[C]fi_endpoint\f[](3), \f[C]fi_domain\f[](3),
\f[C]fi_eq\f[](3)
.SH AUTHORS
OpenFabrics.