ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
f9c65a701e
openmpi/opal/mca/btl/usnic/help-mpi-btl-usnic.txt

331 строка
11 KiB
Plaintext
Исходник Обычный вид История

First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
# -*- text -*-
#
usnic: improve interface matching (esp. for UDP) Prior to this commit we matched local interfaces to remote interfaces in order to create endpoints in a simplistic way. If any remote interfaces were on the same subnet as any of our local interfaces then only local interfaces would be paired (IP-routed remote interfaces would be ignored). This commit introduces a more general scheme which attempts to make the "best" pairing of local interfaces to remote interfaces. We now cast the problem as a graph theory problem known as the "Assignment Problem", or finding a maximum-cardinality, minimum-weight bipartite matching. We solve this problem by reducing the bipartite graph of interface connectivity to a flow network and then solving for a minimum cost flow. This is then easily converted into back into a matching on the original bipartite graph. In the new scheme, interfaces on the same subnet are preferred over interfaces requiring intermediate routing hops and higher bandwidth links are preferred over lower bandwidth links. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30849. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-26 11:50:26 +04:00
# Copyright (c) 2012-2014 Cisco Systems, Inc. All rights reserved.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
#
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
# This is the US/English help file for the Open MPI usnic BTL.
#
[not enough usnic resources]
There are not enough usNIC resources on a VIC for all the MPI
processes on this server.
This means that you have either not provisioned enough usNICs on this
VIC, or there are not enough total receive, transmit, or completion
queues on the provisioned usNICs. On each VIC in a given server, you
need to provision at least as many usNICs as MPI processes on that
server. In each usNIC, you need to provision at least two each of the
following: send queues, receive queues, and completion queues.
usnic: remove "device" and "port" language from show_help() messages Move away from verbs-specific terms "device" and "port" in the usnic BTL help messages. Replace them with "usNIC interface" (since usNIC has no concept of a port). cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32029. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-18 19:20:50 +04:00
Open MPI will skip this usNIC interface in the usnic BTL, which may
result in either lower performance or your job aborting.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Server: %s
usnic: remove "device" and "port" language from show_help() messages Move away from verbs-specific terms "device" and "port" in the usnic BTL help messages. Replace them with "usNIC interface" (since usNIC has no concept of a port). cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32029. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-18 19:20:50 +04:00
usNIC interface: %s
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Description: %s
#
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
[libfabric API failed]
Open MPI failed a basic API operation on a Cisco usNIC interface.
This is highly unusual and shouldn't happen. It suggests that there
may be something wrong with the usNIC configuration on this server.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
In addition to any suggestions listed below, you might want to check
the Linux "memlock" limits on your system (they should probably be
"unlimited"). See this FAQ entry for details:
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Open MPI will skip this usNIC interface in the usnic BTL, which may
result in either lower performance or your job aborting.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Server: %s
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
usNIC interface: %s
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Failed function: %s (%s:%d)
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Return status: %d
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Description: %s
#
[async event]
usnic: remove "device" and "port" language from show_help() messages Move away from verbs-specific terms "device" and "port" in the usnic BTL help messages. Replace them with "usNIC interface" (since usNIC has no concept of a port). cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32029. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-18 19:20:50 +04:00
Open MPI detected a fatal error on a usNIC interface. Your MPI job
will now abort; sorry.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Server: %s
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
usNIC interface: %s
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Async event code: %s (%d)
#
[internal error during init]
An internal error has occurred in the Open MPI usNIC BTL. This is
highly unusual and shouldn't happen. It suggests that there may be
something wrong with the usNIC or OpenFabrics configuration on this
server.
usnic: remove "device" and "port" language from show_help() messages Move away from verbs-specific terms "device" and "port" in the usnic BTL help messages. Replace them with "usNIC interface" (since usNIC has no concept of a port). cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32029. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-18 19:20:50 +04:00
Open MPI will skip this usNIC interface in the usnic BTL, which may
result in either lower performance or your job aborting.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Server: %s
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
usNIC interface: %s
Failure at: %s (%s:%d)
Error: %d (%s)
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
#
[internal error after init]
An internal error has occurred in the Open MPI usNIC BTL. This is
highly unusual and shouldn't happen. It suggests that there may be
something wrong with the usNIC or OpenFabrics configuration on this
server.
Cisco CSCuj22803: fixes for Bsend changes required to support MPI_Bsend(). Introduces concept of attaching a buffer to a large segment that the PML can scribble into and we will send from. The reason we don't use a pinned buffer and send directly from that is that usnic_verbs does not (yes) support num_sge>1 for regular sends. This means the data gets copied twice, but that is unavoidable. changed the logic in handle_large_send to be more sensible Incorporated David's review comments This commit was SVN r29184.
2013-09-17 11:27:39 +04:00
Server: %s
Message: %s
File: %s
Line: %d
btl_usnic_util.c:ompi_btl_usnic_util_abort() also passes in the strerror(). This commit was SVN r29188.
2013-09-17 16:35:51 +04:00
Error: %s
#
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
[check_reg_mem_basics fail]
The usNIC BTL failed to initialize while trying to register some
memory. This typically can indicate that the "memlock" limits are set
too low. For most HPC installations, the memlock limits should be set
Fix a collection of bugs found by QA and Coverity, and make some minor improvements: * Fix minor memory leaks during component_init * Ensure that an initialization loop does not underflow an unsigned int * Improve mlock limit checking * Fix set of BTL modules created during component_init when failing to get QP resources or otherwise excluding some (but not all) usnic verbs devices * Fix/improve error messages to be consistent with other Cisco documentation * Randomize the initial sliding window sequence number so that we silently drop incoming frames from previous jobs that still have existant processes in the middle of dying (and are still transmitting) * Ensure we don't break out of add_procs too soon and create an asymetrical view of what interfaces are available This commit was SVN r28975.
2013-08-01 20:56:15 +04:00
to "unlimited". The failure occurred here:
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Local host: %s
Memlock limit: %s
You may need to consult with your system administrator to get this
problem fixed. This FAQ entry on the Open MPI web site may also be
helpful:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
#
[invalid if_inexclude]
WARNING: An invalid value was given for btl_usnic_if_%s. This
value will be ignored.
Local host: %s
Value: %s
Message: %s
#
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
[device present but not up]
Open MPI has found a usNIC device that is present / listed in Linux,
but in a "down" state. It will not be used by this MPI job.
You may wish to check this device, especially if it is unexpectedly
down.
Local server: %s
Device name: %s
#
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
[MTU mismatch]
usnic: remove "device" and "port" language from show_help() messages Move away from verbs-specific terms "device" and "port" in the usnic BTL help messages. Replace them with "usNIC interface" (since usNIC has no concept of a port). cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32029. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-18 19:20:50 +04:00
The MTU does not match on local and remote hosts. All interfaces on
all hosts participating in an MPI job must be configured with the same
MTU. The usNIC interface listed below will not be used to communicate
with this remote host.
Local host: %s
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
usNIC interface: %s
usnic: remove "device" and "port" language from show_help() messages Move away from verbs-specific terms "device" and "port" in the usnic BTL help messages. Replace them with "usNIC interface" (since usNIC has no concept of a port). cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32029. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-18 19:20:50 +04:00
Local MTU: %d
Remote host: %s
Remote MTU: %d
Minor improvements to the usnic BTL: 1. Fix ompi_info memory leak in usnic BTL: do not allocate memory in the component register function, because ompi_info only calls the component register function and then dlclose's the component -- it does not call component finalize. Instead, defer parsing the MCA param (and alloc'ing memory) until the component init function so that any allocated memory can be freed in the component close function. 1. Also add a new check to ensure that we actually have some part numbers to check. Add a show_help message if we don't find any vendor part IDs to check. 1. Add a verbose output if usnic disqualifies itself from selection because THREAD_MULTIPLE was specified. cmr=v1.7.5:reviewer=dgoodell This commit was SVN r30073.
2013-12-24 15:57:35 +04:00
#
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
[connectivity error: small ok, large bad]
The Open MPI usNIC BTL was unable to establish full connectivity
usnic: update help messages These messages were committed in the v1.8 branch in r32341, but were never committed to the trunk (because we were waiting for the OPAL BTL move). This commit brings the trunk and v1.8 help messages in line with each other. This commit was SVN r32445. The following SVN revision numbers were found above: r32341 --> open-mpi/ompi@5e752b4abaa4859a79372dbb435aceceb17a9eba
2014-08-08 00:50:29 +04:00
between at least one pair of interfaces on servers in the MPI job.
Specifically, small UDP messages seem to flow between the interfaces,
but large UDP messages do not.
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
Your MPI job is going to abort now.
Source:
Hostname / IP: %s (%s)
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Host interface: %s
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
Destination:
Hostname / IP: %s (%s)
Small message size: %u
Large message size: %u
Note that this behavior usually indicates that the MTU of some network
usnic: update connectivity checker help message Show an example of using the btl_usnic_connectivity_map option. Also, mention that another reason for the "total connectivity failure" may be due to asymmetric / unexpected routing. Reviewed by Dave Goodell. cmr=v1.8.2:reviewer=ompi-rm1.8 This commit was SVN r32465.
2014-08-08 21:18:29 +04:00
link is too small between these two interfaces.
You should verify that UDP traffic with payloads up to the "large
message size" listed above can flow between these two interfaces. You
should also verify that Open MPI is choosing to pair IP interfaces
consistently. For example:
mpirun --mca btl_usnic_connectivity_map mymap ...
Check the resulting "mymap*" files to see the exact pairing of IP
interfaces. Inconsistent results may be indicative of underlying
network misconfigurations.
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
#
[connectivity error: small bad, large ok]
The Open MPI usNIC BTL was unable to establish full connectivity
usnic: update help messages These messages were committed in the v1.8 branch in r32341, but were never committed to the trunk (because we were waiting for the OPAL BTL move). This commit brings the trunk and v1.8 help messages in line with each other. This commit was SVN r32445. The following SVN revision numbers were found above: r32341 --> open-mpi/ompi@5e752b4abaa4859a79372dbb435aceceb17a9eba
2014-08-08 00:50:29 +04:00
between at least one pair of interfaces on servers in the MPI job.
Specifically, large UDP messages seem to flow between the interfaces,
but small UDP messages do not.
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
Your MPI job is going to abort now.
Source:
Hostname / IP: %s (%s)
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Host interface: %s
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
Destination:
Hostname / IP: %s (%s)
Small message size: %u
Large message size: %u
This is a very strange network error, and should not occur in most
situations. You may be experiencing high amounts of congestion, or
usnic: update connectivity checker help message Show an example of using the btl_usnic_connectivity_map option. Also, mention that another reason for the "total connectivity failure" may be due to asymmetric / unexpected routing. Reviewed by Dave Goodell. cmr=v1.8.2:reviewer=ompi-rm1.8 This commit was SVN r32465.
2014-08-08 21:18:29 +04:00
this may indicate some kind of network misconfiguration.
You should verify that UDP traffic with payloads up to the "large
message size" listed above can flow between these two interfaces. You
should also verify that Open MPI is choosing to pair IP interfaces
consistently. For example:
mpirun --mca btl_usnic_connectivity_map mymap ...
Check the resulting "mymap*" files to see the exact pairing of IP
interfaces. Inconsistent results may be indicative of underlying
network misconfigurations.
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
#
[connectivity error: small bad, large bad]
The Open MPI usNIC BTL was unable to establish any connectivity
usnic: update help messages These messages were committed in the v1.8 branch in r32341, but were never committed to the trunk (because we were waiting for the OPAL BTL move). This commit brings the trunk and v1.8 help messages in line with each other. This commit was SVN r32445. The following SVN revision numbers were found above: r32341 --> open-mpi/ompi@5e752b4abaa4859a79372dbb435aceceb17a9eba
2014-08-08 00:50:29 +04:00
between at least one pair of interfaces on servers in the MPI job.
usnic: update connectivity checker help message Show an example of using the btl_usnic_connectivity_map option. Also, mention that another reason for the "total connectivity failure" may be due to asymmetric / unexpected routing. Reviewed by Dave Goodell. cmr=v1.8.2:reviewer=ompi-rm1.8 This commit was SVN r32465.
2014-08-08 21:18:29 +04:00
This can happen for several reasons, including:
1. No UDP traffic is able to flow between the interfaces listed below.
2. There is asymmetric routing between the interfaces listed below,
leading Open MPI to discard UDP traffic it thinks is from an
unexpected source.
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
Your MPI job is going to abort now.
Source:
Hostname / IP: %s (%s)
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Host interface: %s
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
Destination:
Hostname / IP: %s (%s)
Small message size: %u
Large message size: %u
Note that this behavior usually indicates some kind of network
usnic: update connectivity checker help message Show an example of using the btl_usnic_connectivity_map option. Also, mention that another reason for the "total connectivity failure" may be due to asymmetric / unexpected routing. Reviewed by Dave Goodell. cmr=v1.8.2:reviewer=ompi-rm1.8 This commit was SVN r32465.
2014-08-08 21:18:29 +04:00
misconfiguration.
You should verify that UDP traffic with payloads up to the "large
message size" listed above can flow between these two interfaces. You
should also verify that Open MPI is choosing to pair IP interfaces
consistently. For example:
mpirun --mca btl_usnic_connectivity_map mymap ...
Check the resulting "mymap*" files to see the exact pairing of IP
interfaces. Inconsistent results may be indicative of underlying
network misconfigurations.
This is what r30852 should have been: Consolidate into a single, outter loop of ibv_create_ah() calls Follow on to SVN trunk r30850: consolidate the ibv_create_ah() calls into a single loop, MPI_WAITALL-style. That is, call the (effectively non-blocking) ibv_create_ah() for each endpoint. If we get NULL+EAGAIN, it means that the UDP ARP is still ongoing down in the kernel, so just try again later. We put these all into a single loop because it allows us to parallelize the ARP progress in the kernel. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30879. The following SVN revision numbers were found above: r30850 --> open-mpi/ompi@3641500442c1362e148d0d6272db91d75ec92116 r30852 --> open-mpi/ompi@4e282a329578c9e7296d5a3809fedfa9f83870df The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 21:19:50 +04:00
#
usnic: fix show_help message Rename a few symbols to use libfabric-friendly names. Fix a show_help message when fi_av_insert times out.
2014-12-05 02:19:34 +03:00
[fi_av_insert timeout]
This is what r30852 should have been: Consolidate into a single, outter loop of ibv_create_ah() calls Follow on to SVN trunk r30850: consolidate the ibv_create_ah() calls into a single loop, MPI_WAITALL-style. That is, call the (effectively non-blocking) ibv_create_ah() for each endpoint. If we get NULL+EAGAIN, it means that the UDP ARP is still ongoing down in the kernel, so just try again later. We put these all into a single loop because it allows us to parallelize the ARP progress in the kernel. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30879. The following SVN revision numbers were found above: r30850 --> open-mpi/ompi@3641500442c1362e148d0d6272db91d75ec92116 r30852 --> open-mpi/ompi@4e282a329578c9e7296d5a3809fedfa9f83870df The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 21:19:50 +04:00
The usnic BTL failed to create addresses for remote peers within the
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
specified timeout. This usually means that ARP requests failed to
resolve in time. You may be able to solve the problem by increasing
the usnic BTL's ARP timeout. If that doesn't work, you should
diagnose why ARP replies are apparently not being delivered in a
timely manner.
This is what r30852 should have been: Consolidate into a single, outter loop of ibv_create_ah() calls Follow on to SVN trunk r30850: consolidate the ibv_create_ah() calls into a single loop, MPI_WAITALL-style. That is, call the (effectively non-blocking) ibv_create_ah() for each endpoint. If we get NULL+EAGAIN, it means that the UDP ARP is still ongoing down in the kernel, so just try again later. We put these all into a single loop because it allows us to parallelize the ARP progress in the kernel. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30879. The following SVN revision numbers were found above: r30850 --> open-mpi/ompi@3641500442c1362e148d0d6272db91d75ec92116 r30852 --> open-mpi/ompi@4e282a329578c9e7296d5a3809fedfa9f83870df The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 21:19:50 +04:00
The usNIC interface listed below will be ignored. Your MPI
application will likely either run with degraded performance and/or
abort.
Server: %s
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
usNIC interface: %s
This is what r30852 should have been: Consolidate into a single, outter loop of ibv_create_ah() calls Follow on to SVN trunk r30850: consolidate the ibv_create_ah() calls into a single loop, MPI_WAITALL-style. That is, call the (effectively non-blocking) ibv_create_ah() for each endpoint. If we get NULL+EAGAIN, it means that the UDP ARP is still ongoing down in the kernel, so just try again later. We put these all into a single loop because it allows us to parallelize the ARP progress in the kernel. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30879. The following SVN revision numbers were found above: r30850 --> open-mpi/ompi@3641500442c1362e148d0d6272db91d75ec92116 r30852 --> open-mpi/ompi@4e282a329578c9e7296d5a3809fedfa9f83870df The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 21:19:50 +04:00
Current ARP timeout: %d (btl_usnic_arp_timeout MCA param)
usnic: ensure that procs agree on use_udp value Add the component use_udp value into the modex. If my component's use_udp value doesn't agree with the use_udp value from a peer's modex data, print a helpful message and disqualify the usnic BTL (the usnic BTL will not be used). This prevents accidental customer misconfigurations. Reviewed by Dave Goodell cmr=v1.8.2:reviewer=ompi-rm1.8 This commit was SVN r31689.
2014-05-08 20:43:50 +04:00
#
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
[unreachable peer IP]
usnic: behave better when ibv_create_ah() fails When ibv_create_ah() fails due to an address resolution failure, it really only means that we can't reach that one peer -- so we should just ignore that one peer. If ibv_create_ah() fails for some other reason, then give up on the entire usnic_X device. Change the show_help() message that is displayed when ibv_create_ah() fails due to address resolution failure; indicate that it's likely a routing problem. Also opal_output_verbose() the same info, since show_help() is de-duplicated (and this particular show_help() message can be squelched). Fixes Cisco bugs CSCup35851 and CSCup35872. cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32061. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-21 00:53:50 +04:00
WARNING: Open MPI failed to find a route to a peer IP address via a
specific usNIC interface. This usually indicates a problem in the IP
routing between these peers.
Open MPI will skip this usNIC interface when communicating with that
peer, which may result in lower performance to that peer. It may also
result in your job aborting if there are no other network paths to
that peer.
Note that this error message defaults to only printing for the *first*
pair of MPI process peers that exhibit this problem; this same problem
may exist for other peer pairs, too.
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Local interface: %s:%s (which is %s)
usnic: behave better when ibv_create_ah() fails When ibv_create_ah() fails due to an address resolution failure, it really only means that we can't reach that one peer -- so we should just ignore that one peer. If ibv_create_ah() fails for some other reason, then give up on the entire usnic_X device. Change the show_help() message that is displayed when ibv_create_ah() fails due to address resolution failure; indicate that it's likely a routing problem. Also opal_output_verbose() the same info, since show_help() is de-duplicated (and this particular show_help() message can be squelched). Fixes Cisco bugs CSCup35851 and CSCup35872. cmr=v1.8.2:ticket=trac:4734 This commit was SVN r32061. The following Trac tickets were found above: Ticket 4734 --> https://svn.open-mpi.org/trac/ompi/ticket/4734
2014-06-21 00:53:50 +04:00
Peer: %s:%s
NOTE: You can set the MCA param btl_usnic_show_route_failures to 0 to
disable this warning.
usnic: show_help if we can't create the connectivity map file QA ran across the case where the user can't write to the target directory for the connectivity map file. In this case, we silently continued. They requested that we at least warn in this case. Fixes Cisco bug CSCup62821 Reviewed by Dave Goodell cmr=v1.8.2:reviewer=ompi-rm1.8 This commit was SVN r32283.
2014-07-23 00:50:59 +04:00
#
[cannot write to map file]
WARNING: The usnic BTL was unable to open the requested output
connectivity map file. Your job will continue, but this output
connectivity map file will not be written.
Local server: %s
Output map file: %s
Working directory: %s
Error: %s (%d)
usnic: convert some BTL_ERRORs to more descriptive show_help messages 1. After we receive N abnormally-short messages (meaning: corrupted), print a show_help message about it. N defaults to 25. N can be set to 0 disable the message via btl_usnic_max_short_packets. 1. If we receive a completion error for something other than a receive, display a show_help message. Reviewed by Dave Goodell. CMR'ing to v1.8.3, but it will require a custom patch because of the OMPI->OPAL BTL move. cmr=v1.8.3 This commit was SVN r32522.
2014-08-13 19:01:20 +04:00
#
[received too many short packets]
WARNING: The usnic BTL received a significant number of abnormally
short packets on a single network interface. This may be due to
usnic: remove suggestion of enabling no-drop in the fabric Reviewed by Reese Faucette cmr=v1.8.3:reviewer=ompi-rm1.8 This commit was SVN r32628.
2014-08-29 03:56:56 +04:00
corruption or congestion in the network fabric. It may be useful to
usnic: convert some BTL_ERRORs to more descriptive show_help messages 1. After we receive N abnormally-short messages (meaning: corrupted), print a show_help message about it. N defaults to 25. N can be set to 0 disable the message via btl_usnic_max_short_packets. 1. If we receive a completion error for something other than a receive, display a show_help message. Reviewed by Dave Goodell. CMR'ing to v1.8.3, but it will require a custom patch because of the OMPI->OPAL BTL move. cmr=v1.8.3 This commit was SVN r32522.
2014-08-13 19:01:20 +04:00
run a physical/layer 0 diagnostic.
Your job will continue, but if this poor network behavior continues,
you may experience lower-than-expected performance due to overheads
caused by higher-than-usual retransmission rates (to compensate for
the corrupted received packets).
Local server: %s
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
usNIC interface: %s
usnic: convert some BTL_ERRORs to more descriptive show_help messages 1. After we receive N abnormally-short messages (meaning: corrupted), print a show_help message about it. N defaults to 25. N can be set to 0 disable the message via btl_usnic_max_short_packets. 1. If we receive a completion error for something other than a receive, display a show_help message. Reviewed by Dave Goodell. CMR'ing to v1.8.3, but it will require a custom patch because of the OMPI->OPAL BTL move. cmr=v1.8.3 This commit was SVN r32522.
2014-08-13 19:01:20 +04:00
# of short packets
received so far: %d
You will only receive this warning once per MPI process per job.
If you know that your network environment is lossy/heavily congested
such that short/corrupted packets are expected, you can disable this
warning by setting the btl_usnic_max_short_packets MCA parameter to 0.
#
[non-receive completion error]
WARNING: The usnic BTL has detected an error in the completion of a
non-receive event. This is highly unusual, and may indicate an error
in the usNIC subsystem on this server.
Your MPI job will continue, but you should monitor the job and ensure
that it behaves correctly.
Local server: %s
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
usNIC interface: %s
usnic: convert some BTL_ERRORs to more descriptive show_help messages 1. After we receive N abnormally-short messages (meaning: corrupted), print a show_help message about it. N defaults to 25. N can be set to 0 disable the message via btl_usnic_max_short_packets. 1. If we receive a completion error for something other than a receive, display a show_help message. Reviewed by Dave Goodell. CMR'ing to v1.8.3, but it will require a custom patch because of the OMPI->OPAL BTL move. cmr=v1.8.3 This commit was SVN r32522.
2014-08-13 19:01:20 +04:00
Channel index: %d
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Completion status: %s (%d)
usnic: convert some BTL_ERRORs to more descriptive show_help messages 1. After we receive N abnormally-short messages (meaning: corrupted), print a show_help message about it. N defaults to 25. N can be set to 0 disable the message via btl_usnic_max_short_packets. 1. If we receive a completion error for something other than a receive, display a show_help message. Reviewed by Dave Goodell. CMR'ing to v1.8.3, but it will require a custom patch because of the OMPI->OPAL BTL move. cmr=v1.8.3 This commit was SVN r32522.
2014-08-13 19:01:20 +04:00
Work request ID: %p
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
Opcode: %s (%d)
usnic: convert some BTL_ERRORs to more descriptive show_help messages 1. After we receive N abnormally-short messages (meaning: corrupted), print a show_help message about it. N defaults to 25. N can be set to 0 disable the message via btl_usnic_max_short_packets. 1. If we receive a completion error for something other than a receive, display a show_help message. Reviewed by Dave Goodell. CMR'ing to v1.8.3, but it will require a custom patch because of the OMPI->OPAL BTL move. cmr=v1.8.3 This commit was SVN r32522.
2014-08-13 19:01:20 +04:00
If this error keeps happening, you should contact Cisco technical
support.
usnic: convert from verbs to libfabric (yay!) This commit represents the conversion of the usnic BTL from verbs to libfabric. For the moment, libfabric is embedded in Open MPI (currently in the usnic BTL). This is because the libfabric API is still changing, and also has not yet been released. Ultimately, this embedded copy of libfabric will likely disappear and the usnic BTL will rely on an external installation of libfabric. New configure options: * --with-libfabric: will cause configure to fail if libfabric support cannot be built * --without-libfabric: will prevent libfabric support from being built * --with-libfabric=DIR: use an external libfabric installation * --with-libfabric-libdir=LIBDIR: when paired with --with-libfabric=DIR, use LIBDIR for the libfabric installation library dir The --with-libnl3[-libdir] arguments are now gone.
2014-12-03 00:09:46 +03:00
#
[device present but not up]
Open MPI has found a usNIC device that is present / listed in Linux,
but in a "down" state. It will not be used by this MPI job.
You may wish to check this device, especially if it is unexpectedly
down.
Local server: %s
Device name: %s
#
[transport mismatch]
Open MPI has found two servers with different underlying usNIC
transports. This is an unsupported configuration; all usNIC devices
must have the same underlying transport in order to use the usNIC BTL.
Local server / transport: %s / %s
Remote server / transport: %s / %s
Копировать постоянную ссылку