ports/openmpi
ports/openmpi
1
1
Форкнуть 0
Код Релизы Активность
9e32cb9b60
openmpi/ompi/mca/btl/usnic/help-mpi-btl-usnic.txt

270 строки
8.5 KiB
Plaintext
Исходник Обычный вид История

First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
# -*- text -*-
#
usnic: improve interface matching (esp. for UDP) Prior to this commit we matched local interfaces to remote interfaces in order to create endpoints in a simplistic way. If any remote interfaces were on the same subnet as any of our local interfaces then only local interfaces would be paired (IP-routed remote interfaces would be ignored). This commit introduces a more general scheme which attempts to make the "best" pairing of local interfaces to remote interfaces. We now cast the problem as a graph theory problem known as the "Assignment Problem", or finding a maximum-cardinality, minimum-weight bipartite matching. We solve this problem by reducing the bipartite graph of interface connectivity to a flow network and then solving for a minimum cost flow. This is then easily converted into back into a matching on the original bipartite graph. In the new scheme, interfaces on the same subnet are preferred over interfaces requiring intermediate routing hops and higher bandwidth links are preferred over lower bandwidth links. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30849. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-26 11:50:26 +04:00
# Copyright (c) 2012-2014 Cisco Systems, Inc. All rights reserved.
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
#
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
# This is the US/English help file for the Open MPI usnic BTL.
#
[ibv API failed]
Open MPI failed a basic verbs operation on a Cisco usNIC device. This
is highly unusual and shouldn't happen. It suggests that there may be
something wrong with the usNIC or OpenFabrics configuration on this
server.
In addition to any suggestions listed below, you might want to check
the Linux "memlock" limits on your system (they should probably be
"unlimited"). See this FAQ entry for details:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
Open MPI will skip this device/port in the usnic BTL, which may result
in either lower performance or your job aborting.
Server: %s
Device: %s
Port: %d
Failed function: %s (%s:%d)
Description: %s
#
[not enough usnic resources]
There are not enough usNIC resources on a VIC for all the MPI
processes on this server.
This means that you have either not provisioned enough usNICs on this
VIC, or there are not enough total receive, transmit, or completion
queues on the provisioned usNICs. On each VIC in a given server, you
need to provision at least as many usNICs as MPI processes on that
server. In each usNIC, you need to provision at least two each of the
following: send queues, receive queues, and completion queues.
Open MPI will skip this device in the usnic BTL, which may result in
either lower performance or your job aborting.
Server: %s
Device: %s
Description: %s
#
[create ibv resource failed]
Open MPI failed to allocate a usNIC-related resource on a VIC. This
usually means one of two things:
1. You are running something other than this MPI job on this server
that is consuming usNIC resources.
Fix a collection of bugs found by QA and Coverity, and make some minor improvements: * Fix minor memory leaks during component_init * Ensure that an initialization loop does not underflow an unsigned int * Improve mlock limit checking * Fix set of BTL modules created during component_init when failing to get QP resources or otherwise excluding some (but not all) usnic verbs devices * Fix/improve error messages to be consistent with other Cisco documentation * Randomize the initial sliding window sequence number so that we silently drop incoming frames from previous jobs that still have existant processes in the middle of dying (and are still transmitting) * Ensure we don't break out of add_procs too soon and create an asymetrical view of what interfaces are available This commit was SVN r28975.
2013-08-01 20:56:15 +04:00
2. You have run out of locked Linux memory. You should probably set
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
the Linux "memlock" limits to "unlimited". See this FAQ entry for
details:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
This Open MPI job will skip this device/port in the usnic BTL, which
may result in either lower performance or the job aborting.
Server: %s
Device: %s
Failed function: %s (%s:%d)
Description: %s
#
[async event]
Open MPI detected a fatal error on a usNIC port. Your MPI job will
now abort; sorry.
Server: %s
Device: %s
Port: %d
Async event code: %s (%d)
#
[internal error during init]
An internal error has occurred in the Open MPI usNIC BTL. This is
highly unusual and shouldn't happen. It suggests that there may be
something wrong with the usNIC or OpenFabrics configuration on this
server.
Open MPI will skip this device/port in the usnic BTL, which may result
in either lower performance or your job aborting.
Server: %s
Device: %s
Port: %d
Failure: %s (%s:%d)
Description: %s
#
[internal error after init]
An internal error has occurred in the Open MPI usNIC BTL. This is
highly unusual and shouldn't happen. It suggests that there may be
something wrong with the usNIC or OpenFabrics configuration on this
server.
Cisco CSCuj22803: fixes for Bsend changes required to support MPI_Bsend(). Introduces concept of attaching a buffer to a large segment that the PML can scribble into and we will send from. The reason we don't use a pinned buffer and send directly from that is that usnic_verbs does not (yes) support num_sge>1 for regular sends. This means the data gets copied twice, but that is unavoidable. changed the logic in handle_large_send to be more sensible Incorporated David's review comments This commit was SVN r29184.
2013-09-17 11:27:39 +04:00
Server: %s
Message: %s
File: %s
Line: %d
btl_usnic_util.c:ompi_btl_usnic_util_abort() also passes in the strerror(). This commit was SVN r29188.
2013-09-17 16:35:51 +04:00
Error: %s
#
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
[verbs_port_bw failed]
Open MPI failed to query the supported bandwidth of a port on a Cisco
usNIC device. This is unusual and shouldn't happen. It suggests that
there may be something wrong with the usNIC or OpenFabrics
configuration on this server.
Open MPI will skip this device/port in the usnic BTL, which may result
in either lower performance or your job aborting.
Server: %s
Device: %s
Port: %d
#
[check_reg_mem_basics fail]
The usNIC BTL failed to initialize while trying to register some
memory. This typically can indicate that the "memlock" limits are set
too low. For most HPC installations, the memlock limits should be set
Fix a collection of bugs found by QA and Coverity, and make some minor improvements: * Fix minor memory leaks during component_init * Ensure that an initialization loop does not underflow an unsigned int * Improve mlock limit checking * Fix set of BTL modules created during component_init when failing to get QP resources or otherwise excluding some (but not all) usnic verbs devices * Fix/improve error messages to be consistent with other Cisco documentation * Randomize the initial sliding window sequence number so that we silently drop incoming frames from previous jobs that still have existant processes in the middle of dying (and are still transmitting) * Ensure we don't break out of add_procs too soon and create an asymetrical view of what interfaces are available This commit was SVN r28975.
2013-08-01 20:56:15 +04:00
to "unlimited". The failure occurred here:
First commit of the Cisco usNIC BTL. This BTL accesses the Cisco usNIC Linux device via the Linux verbs API via Unreliable Datagram queue pairs. A few noteworthy points: * This BTL does most of its own fragmentation; it tells the PML that it has a very high max_send_size (much higher than the network MTU). * Since UD fragments are, by definition, unreliable, the usnic BTL handles all of its own reliability via a sliding window approach using the opal_hotel construct and many tricks stolen from the corpus of knowledge surrounding efficient TCP. * There is a fun PML latency-metric based optimization for NUMA awareness of short messages. * Note that this is ''not'' a generic UD verbs BTL; it is specific to the Cisco usNIC device. This commit was SVN r28879.
2013-07-20 02:13:58 +04:00
Local host: %s
Memlock limit: %s
You may need to consult with your system administrator to get this
problem fixed. This FAQ entry on the Open MPI web site may also be
helpful:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
#
[invalid if_inexclude]
WARNING: An invalid value was given for btl_usnic_if_%s. This
value will be ignored.
Local host: %s
Value: %s
Message: %s
#
[MTU mismatch]
The MTU does not match on local and remote hosts. All interfaces on all
hosts participating in an MPI job must be configured with the same MTU.
The device and port listed below will not be used to communicate with this
remote host.
Local host: %s
Device/port: %s/%d
Local MTU: %d
Remote host: %s
Remote MTU: %d
Minor improvements to the usnic BTL: 1. Fix ompi_info memory leak in usnic BTL: do not allocate memory in the component register function, because ompi_info only calls the component register function and then dlclose's the component -- it does not call component finalize. Instead, defer parsing the MCA param (and alloc'ing memory) until the component init function so that any allocated memory can be freed in the component close function. 1. Also add a new check to ensure that we actually have some part numbers to check. Add a show_help message if we don't find any vendor part IDs to check. 1. Add a verbose output if usnic disqualifies itself from selection because THREAD_MULTIPLE was specified. cmr=v1.7.5:reviewer=dgoodell This commit was SVN r30073.
2013-12-24 15:57:35 +04:00
#
usnic: improve interface matching (esp. for UDP) Prior to this commit we matched local interfaces to remote interfaces in order to create endpoints in a simplistic way. If any remote interfaces were on the same subnet as any of our local interfaces then only local interfaces would be paired (IP-routed remote interfaces would be ignored). This commit introduces a more general scheme which attempts to make the "best" pairing of local interfaces to remote interfaces. We now cast the problem as a graph theory problem known as the "Assignment Problem", or finding a maximum-cardinality, minimum-weight bipartite matching. We solve this problem by reducing the bipartite graph of interface connectivity to a flow network and then solving for a minimum cost flow. This is then easily converted into back into a matching on the original bipartite graph. In the new scheme, interfaces on the same subnet are preferred over interfaces requiring intermediate routing hops and higher bandwidth links are preferred over lower bandwidth links. Reviewed-by: Jeff Squyres <jsquyres@cisco.com> cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30849. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-26 11:50:26 +04:00
[rtnetlink init fail]
The usnic BTL failed to initialize the rtnetlink query subsystem.
Server: %s
Error message: %s
Add usnic connectivity-checking agent service. Basically: since usnic is a connectionless transport, we do not get OS-provided services "for free" that connection-oriented transports get, namely: "hey, I wasn't able to make a connection to peer X", and "hey, your connection to peer X has died." This connectivity-checker runs in a separate progress thread in the usnic BTL in local rank 0 on each server. Upon first send in any process, the connectivty-checker agent will send some UDP pings to the peer to ensure that we can reach it. If we can't, we'll abort the job with a nice show_help message. There's a lengthy comment in btl_usnic_connectivity.h explains the scheme and how it works. Reviewed by Dave Goodell. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30860. The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 02:21:25 +04:00
#
[connectivity error: small ok, large bad]
The Open MPI usNIC BTL was unable to establish full connectivity
between at least one pair of servers in the MPI job. Specifically,
small UDP messages seem to flow between the servers, but large UDP
messages do not.
Your MPI job is going to abort now.
Source:
Hostname / IP: %s (%s)
Host interfaces: %s / %s
MAC address: %s
Destination:
Hostname / IP: %s (%s)
MAC address: %s
Small message size: %u
Large message size: %u
Note that this behavior usually indicates that the MTU of some network
link is too small between these two servers. You should verify that
UDP traffic with payloads up to the "large message size" listed above
can flow between these two servers.
#
[connectivity error: small bad, large ok]
The Open MPI usNIC BTL was unable to establish full connectivity
between at least one pair of servers in the MPI job. Specifically,
large UDP messages seem to flow between the servers, but small UDP
messages do not.
Your MPI job is going to abort now.
Source:
Hostname / IP: %s (%s)
Host interfaces: %s / %s
MAC address: %s
Destination:
Hostname / IP: %s (%s)
MAC address: %s
Small message size: %u
Large message size: %u
This is a very strange network error, and should not occur in most
situations. You may be experiencing high amounts of congestion, or
this may indicate some kind of network misconfiguration. You should
verify that UDP traffic with payloads up to the "large message size"
listed above can flow between these two servers.
#
[connectivity error: small bad, large bad]
The Open MPI usNIC BTL was unable to establish any connectivity
between at least one pair of servers in the MPI job. Specifically,
no UDP messages seemed to flow between these two servers.
Your MPI job is going to abort now.
Source:
Hostname / IP: %s (%s)
Host interfaces: %s / %s
MAC address: %s
Destination:
Hostname / IP: %s (%s)
MAC address: %s
Small message size: %u
Large message size: %u
Note that this behavior usually indicates some kind of network
misconfiguration. You should verify that UDP traffic with payloads up
to the "large message size" listed above can flow between these two
servers.
This is what r30852 should have been: Consolidate into a single, outter loop of ibv_create_ah() calls Follow on to SVN trunk r30850: consolidate the ibv_create_ah() calls into a single loop, MPI_WAITALL-style. That is, call the (effectively non-blocking) ibv_create_ah() for each endpoint. If we get NULL+EAGAIN, it means that the UDP ARP is still ongoing down in the kernel, so just try again later. We put these all into a single loop because it allows us to parallelize the ARP progress in the kernel. cmr=v1.7.5:ticket=trac:4253 This commit was SVN r30879. The following SVN revision numbers were found above: r30850 --> open-mpi/ompi@3641500442c1362e148d0d6272db91d75ec92116 r30852 --> open-mpi/ompi@4e282a329578c9e7296d5a3809fedfa9f83870df The following Trac tickets were found above: Ticket 4253 --> https://svn.open-mpi.org/trac/ompi/ticket/4253
2014-02-27 21:19:50 +04:00
#
[ibv_create_ah timeout]
The usnic BTL failed to create addresses for remote peers within the
specified timeout. When using the usNIC/UDP transport, this usually
means that ARP requests failed to resolve in time. You may be able to
solve the problem by increasing the usnic BTL's ARP timeout. If that
doesn't work, you should diagnose why ARP replies are apparently not
being delivered in a timely manner.
The usNIC interface listed below will be ignored. Your MPI
application will likely either run with degraded performance and/or
abort.
Server: %s
Device: %s:%d (%s)
Current ARP timeout: %d (btl_usnic_arp_timeout MCA param)
usnic: ensure that procs agree on use_udp value Add the component use_udp value into the modex. If my component's use_udp value doesn't agree with the use_udp value from a peer's modex data, print a helpful message and disqualify the usnic BTL (the usnic BTL will not be used). This prevents accidental customer misconfigurations. Reviewed by Dave Goodell cmr=v1.8.2:reviewer=ompi-rm1.8 This commit was SVN r31689.
2014-05-08 20:43:50 +04:00
#
[transport mismatch]
The underlying transports used by the usNIC driver stack on multiple
servers do not match. This configuration is unsupported and is almost
certainly not what you want.
This error indicates that the VIC firmware, Linux usNIC kernel driver,
and/or Linux usNIC userspace drivers are not compatible between at
least the following two servers:
Local server: %s
Remote server: %s
The usnic MPI transport will be deactivated in at least the one local
MPI process that reported the problem. This may lead to performance
degradation, and may also result in aborting the overall MPI job.
It is usually easiest to have the same VIC firmware, Linux usNIC
kernel driver, and Linux usNIC userspace driver installed on all
servers.
#
Копировать постоянную ссылку