|
|
|
|
@ -1,34 +1,33 @@
|
|
|
|
|
Introduction
|
|
|
|
|
|
|
|
|
|
hwloc provides command line tools and a C API to obtain the
|
|
|
|
|
hierarchical map of key computing elements, such as: NUMA memory nodes,
|
|
|
|
|
shared caches, processor packages, processor cores, processing units
|
|
|
|
|
(logical processors or "threads") and even I/O devices. hwloc also
|
|
|
|
|
gathers various attributes such as cache and memory information, and is
|
|
|
|
|
portable across a variety of different operating systems and platforms.
|
|
|
|
|
Additionally it may assemble the topologies of multiple machines into a
|
|
|
|
|
single one so as to let applications consult the topology of an entire
|
|
|
|
|
fabric or cluster at once.
|
|
|
|
|
hwloc provides command line tools and a C API to obtain the hierarchical map of
|
|
|
|
|
key computing elements, such as: NUMA memory nodes, shared caches, processor
|
|
|
|
|
packages, processor cores, processing units (logical processors or "threads")
|
|
|
|
|
and even I/O devices. hwloc also gathers various attributes such as cache and
|
|
|
|
|
memory information, and is portable across a variety of different operating
|
|
|
|
|
systems and platforms. Additionally it may assemble the topologies of multiple
|
|
|
|
|
machines into a single one so as to let applications consult the topology of an
|
|
|
|
|
entire fabric or cluster at once.
|
|
|
|
|
|
|
|
|
|
hwloc primarily aims at helping high-performance computing (HPC)
|
|
|
|
|
applications, but is also applicable to any project seeking to exploit
|
|
|
|
|
code and/or data locality on modern computing platforms.
|
|
|
|
|
hwloc primarily aims at helping high-performance computing (HPC) applications,
|
|
|
|
|
but is also applicable to any project seeking to exploit code and/or data
|
|
|
|
|
locality on modern computing platforms.
|
|
|
|
|
|
|
|
|
|
Note that the hwloc project represents the merger of the libtopology
|
|
|
|
|
project from inria and the Portable Linux Processor Affinity (PLPA)
|
|
|
|
|
sub-project from Open MPI. Both of these prior projects are now
|
|
|
|
|
deprecated. The first hwloc release was essentially a "re-branding" of
|
|
|
|
|
the libtopology code base, but with both a few genuinely new features
|
|
|
|
|
and a few PLPA-like features added in. Prior releases of hwloc included
|
|
|
|
|
documentation about switching from PLPA to hwloc; this documentation
|
|
|
|
|
has been dropped on the assumption that everyone who was using PLPA has
|
|
|
|
|
already switched to hwloc.
|
|
|
|
|
Note that the hwloc project represents the merger of the libtopology project
|
|
|
|
|
from inria and the Portable Linux Processor Affinity (PLPA) sub-project from
|
|
|
|
|
Open MPI. Both of these prior projects are now deprecated. The first hwloc
|
|
|
|
|
release was essentially a "re-branding" of the libtopology code base, but with
|
|
|
|
|
both a few genuinely new features and a few PLPA-like features added in. Prior
|
|
|
|
|
releases of hwloc included documentation about switching from PLPA to hwloc;
|
|
|
|
|
this documentation has been dropped on the assumption that everyone who was
|
|
|
|
|
using PLPA has already switched to hwloc.
|
|
|
|
|
|
|
|
|
|
hwloc supports the following operating systems:
|
|
|
|
|
* Linux (including old kernels not having sysfs topology information,
|
|
|
|
|
with knowledge of cpusets, offline CPUs, ScaleMP vSMP, NumaScale
|
|
|
|
|
NumaConnect, and Kerrighed support) on all supported hardware,
|
|
|
|
|
including Intel Xeon Phi (either standalone or as a coprocessor).
|
|
|
|
|
|
|
|
|
|
* Linux (including old kernels not having sysfs topology information, with
|
|
|
|
|
knowledge of cpusets, offline CPUs, ScaleMP vSMP, NumaScale NumaConnect,
|
|
|
|
|
and Kerrighed support) on all supported hardware, including Intel Xeon Phi
|
|
|
|
|
(either standalone or as a coprocessor).
|
|
|
|
|
* Solaris
|
|
|
|
|
* AIX
|
|
|
|
|
* Darwin / OS X
|
|
|
|
|
@ -39,127 +38,126 @@ hwloc supports the following operating systems:
|
|
|
|
|
* Microsoft Windows
|
|
|
|
|
* IBM BlueGene/Q Compute Node Kernel (CNK)
|
|
|
|
|
|
|
|
|
|
Since it uses standard Operating System information, hwloc's support is
|
|
|
|
|
mostly independant from the processor type (x86, powerpc, ...) and just
|
|
|
|
|
relies on the Operating System support. The only exception to this is
|
|
|
|
|
kFreeBSD, which does not support topology information, and hwloc thus
|
|
|
|
|
uses an x86-only CPUID-based backend (which can be used for other OSes
|
|
|
|
|
too, see the Components and plugins section).
|
|
|
|
|
Since it uses standard Operating System information, hwloc's support is mostly
|
|
|
|
|
independant from the processor type (x86, powerpc, ...) and just relies on the
|
|
|
|
|
Operating System support. The only exception to this is kFreeBSD, which does
|
|
|
|
|
not support topology information, and hwloc thus uses an x86-only CPUID-based
|
|
|
|
|
backend (which can be used for other OSes too, see the Components and plugins
|
|
|
|
|
section).
|
|
|
|
|
|
|
|
|
|
To check whether hwloc works on a particular machine, just try to build
|
|
|
|
|
it and run lstopo or lstopo-no-graphics. If some things do not look
|
|
|
|
|
right (e.g. bogus or missing cache information), see Questions and Bugs
|
|
|
|
|
below.
|
|
|
|
|
To check whether hwloc works on a particular machine, just try to build it and
|
|
|
|
|
run lstopo or lstopo-no-graphics. If some things do not look right (e.g. bogus
|
|
|
|
|
or missing cache information), see Questions and Bugs below.
|
|
|
|
|
|
|
|
|
|
hwloc only reports the number of processors on unsupported operating
|
|
|
|
|
systems; no topology information is available.
|
|
|
|
|
hwloc only reports the number of processors on unsupported operating systems;
|
|
|
|
|
no topology information is available.
|
|
|
|
|
|
|
|
|
|
For development and debugging purposes, hwloc also offers the ability
|
|
|
|
|
to work on "fake" topologies:
|
|
|
|
|
* Symmetrical tree of resources generated from a list of level
|
|
|
|
|
arities
|
|
|
|
|
* Remote machine simulation through the gathering of Linux sysfs
|
|
|
|
|
topology files
|
|
|
|
|
For development and debugging purposes, hwloc also offers the ability to work
|
|
|
|
|
on "fake" topologies:
|
|
|
|
|
|
|
|
|
|
hwloc can display the topology in a human-readable format, either in
|
|
|
|
|
graphical mode (X11), or by exporting in one of several different
|
|
|
|
|
formats, including: plain text, PDF, PNG, and FIG (see CLI Examples
|
|
|
|
|
below). Note that some of the export formats require additional support
|
|
|
|
|
libraries.
|
|
|
|
|
* Symmetrical tree of resources generated from a list of level arities
|
|
|
|
|
* Remote machine simulation through the gathering of Linux sysfs topology
|
|
|
|
|
files
|
|
|
|
|
|
|
|
|
|
hwloc offers a programming interface for manipulating topologies and
|
|
|
|
|
objects. It also brings a powerful CPU bitmap API that is used to
|
|
|
|
|
describe topology objects location on physical/logical processors. See
|
|
|
|
|
the Programming Interface below. It may also be used to binding
|
|
|
|
|
applications onto certain cores or memory nodes. Several utility
|
|
|
|
|
programs are also provided to ease command-line manipulation of
|
|
|
|
|
topology objects, binding of processes, and so on.
|
|
|
|
|
hwloc can display the topology in a human-readable format, either in graphical
|
|
|
|
|
mode (X11), or by exporting in one of several different formats, including:
|
|
|
|
|
plain text, PDF, PNG, and FIG (see CLI Examples below). Note that some of the
|
|
|
|
|
export formats require additional support libraries.
|
|
|
|
|
|
|
|
|
|
hwloc offers a programming interface for manipulating topologies and objects.
|
|
|
|
|
It also brings a powerful CPU bitmap API that is used to describe topology
|
|
|
|
|
objects location on physical/logical processors. See the Programming Interface
|
|
|
|
|
below. It may also be used to binding applications onto certain cores or memory
|
|
|
|
|
nodes. Several utility programs are also provided to ease command-line
|
|
|
|
|
manipulation of topology objects, binding of processes, and so on.
|
|
|
|
|
|
|
|
|
|
Perl bindings are available from Bernd Kallies on CPAN.
|
|
|
|
|
|
|
|
|
|
Python bindings are available from Guy Streeter:
|
|
|
|
|
|
|
|
|
|
* Fedora RPM and tarball.
|
|
|
|
|
* git tree (html).
|
|
|
|
|
|
|
|
|
|
Installation
|
|
|
|
|
|
|
|
|
|
hwloc (http://www.open-mpi.org/projects/hwloc/) is available under the
|
|
|
|
|
BSD license. It is hosted as a sub-project of the overall Open MPI
|
|
|
|
|
project (http://www.open-mpi.org/). Note that hwloc does not require
|
|
|
|
|
any functionality from Open MPI -- it is a wholly separate (and much
|
|
|
|
|
smaller!) project and code base. It just happens to be hosted as part
|
|
|
|
|
of the overall Open MPI project.
|
|
|
|
|
hwloc (http://www.open-mpi.org/projects/hwloc/) is available under the BSD
|
|
|
|
|
license. It is hosted as a sub-project of the overall Open MPI project (http://
|
|
|
|
|
www.open-mpi.org/). Note that hwloc does not require any functionality from
|
|
|
|
|
Open MPI -- it is a wholly separate (and much smaller!) project and code base.
|
|
|
|
|
It just happens to be hosted as part of the overall Open MPI project.
|
|
|
|
|
|
|
|
|
|
Nightly development snapshots are available on the web site. Additionally, the
|
|
|
|
|
code can be directly cloned from Git:
|
|
|
|
|
|
|
|
|
|
Nightly development snapshots are available on the web site.
|
|
|
|
|
Additionally, the code can be directly cloned from Git:
|
|
|
|
|
shell$ git clone https://github.com/open-mpi/hwloc.git
|
|
|
|
|
shell$ cd hwloc
|
|
|
|
|
shell$ ./autogen.sh
|
|
|
|
|
|
|
|
|
|
Note that GNU Autoconf >=2.63, Automake >=1.10 and Libtool >=2.2.6 are
|
|
|
|
|
required when building from a Git clone.
|
|
|
|
|
Note that GNU Autoconf >=2.63, Automake >=1.10 and Libtool >=2.2.6 are required
|
|
|
|
|
when building from a Git clone.
|
|
|
|
|
|
|
|
|
|
Installation by itself is the fairly common GNU-based process:
|
|
|
|
|
|
|
|
|
|
shell$ ./configure --prefix=...
|
|
|
|
|
shell$ make
|
|
|
|
|
shell$ make install
|
|
|
|
|
|
|
|
|
|
The hwloc command-line tool "lstopo" produces human-readable topology
|
|
|
|
|
maps, as mentioned above. It can also export maps to the "fig" file
|
|
|
|
|
format. Support for PDF, Postscript, and PNG exporting is provided if
|
|
|
|
|
the "Cairo" development package (usually cairo-devel or libcairo2-dev)
|
|
|
|
|
can be found in "lstopo" when hwloc is configured and build.
|
|
|
|
|
The hwloc command-line tool "lstopo" produces human-readable topology maps, as
|
|
|
|
|
mentioned above. It can also export maps to the "fig" file format. Support for
|
|
|
|
|
PDF, Postscript, and PNG exporting is provided if the "Cairo" development
|
|
|
|
|
package (usually cairo-devel or libcairo2-dev) can be found in "lstopo" when
|
|
|
|
|
hwloc is configured and build.
|
|
|
|
|
|
|
|
|
|
The hwloc core may also benefit from the following development
|
|
|
|
|
packages:
|
|
|
|
|
* libnuma for memory binding and migration support on Linux
|
|
|
|
|
(numactl-devel or libnuma-dev package).
|
|
|
|
|
The hwloc core may also benefit from the following development packages:
|
|
|
|
|
|
|
|
|
|
* libnuma for memory binding and migration support on Linux (numactl-devel or
|
|
|
|
|
libnuma-dev package).
|
|
|
|
|
* libpciaccess for full I/O device discovery (libpciaccess-devel or
|
|
|
|
|
libpciaccess-dev package). On Linux, PCI discovery may still be
|
|
|
|
|
performed (without vendor/device names) even if libpciaccess cannot
|
|
|
|
|
be used.
|
|
|
|
|
libpciaccess-dev package). On Linux, PCI discovery may still be performed
|
|
|
|
|
(without vendor/device names) even if libpciaccess cannot be used.
|
|
|
|
|
|
|
|
|
|
* the AMD OpenCL implementation for OpenCL device discovery.
|
|
|
|
|
* the NVIDIA CUDA Toolkit for CUDA device discovery.
|
|
|
|
|
* the NVIDIA Tesla Development Kit for NVML device discovery.
|
|
|
|
|
* the NV-CONTROL X extension library (NVCtrl) for NVIDIA display
|
|
|
|
|
discovery.
|
|
|
|
|
* the NV-CONTROL X extension library (NVCtrl) for NVIDIA display discovery.
|
|
|
|
|
* libxml2 for full XML import/export support (otherwise, the internal
|
|
|
|
|
minimalistic parser will only be able to import XML files that were
|
|
|
|
|
exported by the same hwloc release). See Importing and exporting
|
|
|
|
|
topologies from/to XML files for details. The relevant development
|
|
|
|
|
package is usually libxml2-devel or libxml2-dev.
|
|
|
|
|
* libudev on Linux for easier discovery of OS device information
|
|
|
|
|
(otherwise hwloc will try to manually parse udev raw files). The
|
|
|
|
|
relevant development package is usually libudev-devel or
|
|
|
|
|
libudev-dev.
|
|
|
|
|
* libtool's ltdl library for dynamic plugin loading. The relevant
|
|
|
|
|
development package is usually libtool-ltdl-devel or libltdl-dev.
|
|
|
|
|
minimalistic parser will only be able to import XML files that were
|
|
|
|
|
exported by the same hwloc release). See Importing and exporting topologies
|
|
|
|
|
from/to XML files for details. The relevant development package is usually
|
|
|
|
|
libxml2-devel or libxml2-dev.
|
|
|
|
|
* libudev on Linux for easier discovery of OS device information (otherwise
|
|
|
|
|
hwloc will try to manually parse udev raw files). The relevant development
|
|
|
|
|
package is usually libudev-devel or libudev-dev.
|
|
|
|
|
* libtool's ltdl library for dynamic plugin loading. The relevant development
|
|
|
|
|
package is usually libtool-ltdl-devel or libltdl-dev.
|
|
|
|
|
|
|
|
|
|
PCI and XML support may be statically built inside the main hwloc
|
|
|
|
|
library, or as separate dynamically-loaded plugins (see the Components
|
|
|
|
|
and plugins section).
|
|
|
|
|
PCI and XML support may be statically built inside the main hwloc library, or
|
|
|
|
|
as separate dynamically-loaded plugins (see the Components and plugins
|
|
|
|
|
section).
|
|
|
|
|
|
|
|
|
|
Note that because of the possibility of GPL taint, the pciutils library
|
|
|
|
|
libpci will not be used (remember that hwloc is BSD-licensed).
|
|
|
|
|
Note that because of the possibility of GPL taint, the pciutils library libpci
|
|
|
|
|
will not be used (remember that hwloc is BSD-licensed).
|
|
|
|
|
|
|
|
|
|
Also note that if you install supplemental libraries in non-standard
|
|
|
|
|
locations, hwloc's configure script may not be able to find them
|
|
|
|
|
without some help. You may need to specify additional CPPFLAGS,
|
|
|
|
|
LDFLAGS, or PKG_CONFIG_PATH values on the configure command line.
|
|
|
|
|
Also note that if you install supplemental libraries in non-standard locations,
|
|
|
|
|
hwloc's configure script may not be able to find them without some help. You
|
|
|
|
|
may need to specify additional CPPFLAGS, LDFLAGS, or PKG_CONFIG_PATH values on
|
|
|
|
|
the configure command line.
|
|
|
|
|
|
|
|
|
|
For example, if libpciaccess was installed into /opt/pciaccess, hwloc's
|
|
|
|
|
configure script may not find it be default. Try adding PKG_CONFIG_PATH
|
|
|
|
|
to the ./configure command line, like this:
|
|
|
|
|
configure script may not find it be default. Try adding PKG_CONFIG_PATH to the
|
|
|
|
|
./configure command line, like this:
|
|
|
|
|
|
|
|
|
|
./configure PKG_CONFIG_PATH=/opt/pciaccess/lib/pkgconfig ...
|
|
|
|
|
|
|
|
|
|
CLI Examples
|
|
|
|
|
|
|
|
|
|
On a 4-package 2-core machine with hyper-threading, the lstopo tool may
|
|
|
|
|
show the following graphical output:
|
|
|
|
|
On a 4-package 2-core machine with hyper-threading, the lstopo tool may show
|
|
|
|
|
the following graphical output:
|
|
|
|
|
|
|
|
|
|
dudley.png
|
|
|
|
|
dudley.png
|
|
|
|
|
|
|
|
|
|
Here's the equivalent output in textual form:
|
|
|
|
|
|
|
|
|
|
Machine (16GB)
|
|
|
|
|
Package L#0 + L3 L#0 (4096KB)
|
|
|
|
|
L2 L#0 (1024KB) + L1 L#0 (16KB) + Core L#0
|
|
|
|
|
@ -190,17 +188,17 @@ Machine (16GB)
|
|
|
|
|
PU L#14 (P#7)
|
|
|
|
|
PU L#15 (P#15)
|
|
|
|
|
|
|
|
|
|
Note that there is also an equivalent output in XML that is meant for
|
|
|
|
|
exporting/importing topologies but it is hardly readable to
|
|
|
|
|
human-beings (see Importing and exporting topologies from/to XML files
|
|
|
|
|
for details).
|
|
|
|
|
Note that there is also an equivalent output in XML that is meant for exporting
|
|
|
|
|
/importing topologies but it is hardly readable to human-beings (see Importing
|
|
|
|
|
and exporting topologies from/to XML files for details).
|
|
|
|
|
|
|
|
|
|
On a 4-package 2-core Opteron NUMA machine, the lstopo tool may show
|
|
|
|
|
the following graphical output:
|
|
|
|
|
On a 4-package 2-core Opteron NUMA machine, the lstopo tool may show the
|
|
|
|
|
following graphical output:
|
|
|
|
|
|
|
|
|
|
hagrid.png
|
|
|
|
|
hagrid.png
|
|
|
|
|
|
|
|
|
|
Here's the equivalent output in textual form:
|
|
|
|
|
|
|
|
|
|
Machine (32GB)
|
|
|
|
|
NUMANode L#0 (P#0 8190MB) + Package L#0
|
|
|
|
|
L2 L#0 (1024KB) + L1 L#0 (64KB) + Core L#0 + PU L#0 (P#0)
|
|
|
|
|
@ -215,12 +213,13 @@ Machine (32GB)
|
|
|
|
|
L2 L#6 (1024KB) + L1 L#6 (64KB) + Core L#6 + PU L#6 (P#6)
|
|
|
|
|
L2 L#7 (1024KB) + L1 L#7 (64KB) + Core L#7 + PU L#7 (P#7)
|
|
|
|
|
|
|
|
|
|
On a 2-package quad-core Xeon (pre-Nehalem, with 2 dual-core dies into
|
|
|
|
|
each package):
|
|
|
|
|
On a 2-package quad-core Xeon (pre-Nehalem, with 2 dual-core dies into each
|
|
|
|
|
package):
|
|
|
|
|
|
|
|
|
|
emmett.png
|
|
|
|
|
emmett.png
|
|
|
|
|
|
|
|
|
|
Here's the same output in textual form:
|
|
|
|
|
|
|
|
|
|
Machine (16GB)
|
|
|
|
|
Package L#0
|
|
|
|
|
L2 L#0 (4096KB)
|
|
|
|
|
@ -239,117 +238,290 @@ Machine (16GB)
|
|
|
|
|
|
|
|
|
|
Programming Interface
|
|
|
|
|
|
|
|
|
|
The basic interface is available in hwloc.h. Some higher-level
|
|
|
|
|
functions are available in hwloc/helper.h to reduce the need to
|
|
|
|
|
manually manipulate objects and follow links between them.
|
|
|
|
|
Documentation for all these is provided later in this document.
|
|
|
|
|
Developers may also want to look at hwloc/inlines.h which contains the
|
|
|
|
|
actual inline code of some hwloc.h routines, and at this document,
|
|
|
|
|
The basic interface is available in hwloc.h. Some higher-level functions are
|
|
|
|
|
available in hwloc/helper.h to reduce the need to manually manipulate objects
|
|
|
|
|
and follow links between them. Documentation for all these is provided later in
|
|
|
|
|
this document. Developers may also want to look at hwloc/inlines.h which
|
|
|
|
|
contains the actual inline code of some hwloc.h routines, and at this document,
|
|
|
|
|
which provides good higher-level topology traversal examples.
|
|
|
|
|
|
|
|
|
|
To precisely define the vocabulary used by hwloc, a Terms and
|
|
|
|
|
Definitions section is available and should probably be read first.
|
|
|
|
|
To precisely define the vocabulary used by hwloc, a Terms and Definitions
|
|
|
|
|
section is available and should probably be read first.
|
|
|
|
|
|
|
|
|
|
Each hwloc object contains a cpuset describing the list of processing
|
|
|
|
|
units that it contains. These bitmaps may be used for CPU binding and
|
|
|
|
|
Memory binding. hwloc offers an extensive bitmap manipulation interface
|
|
|
|
|
in hwloc/bitmap.h.
|
|
|
|
|
Each hwloc object contains a cpuset describing the list of processing units
|
|
|
|
|
that it contains. These bitmaps may be used for CPU binding and Memory binding.
|
|
|
|
|
hwloc offers an extensive bitmap manipulation interface in hwloc/bitmap.h.
|
|
|
|
|
|
|
|
|
|
Moreover, hwloc also comes with additional helpers for interoperability
|
|
|
|
|
with several commonly used environments. See the Interoperability With
|
|
|
|
|
Other Software section for details.
|
|
|
|
|
Moreover, hwloc also comes with additional helpers for interoperability with
|
|
|
|
|
several commonly used environments. See the Interoperability With Other
|
|
|
|
|
Software section for details.
|
|
|
|
|
|
|
|
|
|
The complete API documentation is available in a full set of HTML
|
|
|
|
|
pages, man pages, and self-contained PDF files (formatted for both both
|
|
|
|
|
US letter and A4 formats) in the source tarball in doc/doxygen-doc/.
|
|
|
|
|
The complete API documentation is available in a full set of HTML pages, man
|
|
|
|
|
pages, and self-contained PDF files (formatted for both both US letter and A4
|
|
|
|
|
formats) in the source tarball in doc/doxygen-doc/.
|
|
|
|
|
|
|
|
|
|
NOTE: If you are building the documentation from a Git clone, you will
|
|
|
|
|
need to have Doxygen and pdflatex installed -- the documentation will
|
|
|
|
|
be built during the normal "make" process. The documentation is
|
|
|
|
|
installed during "make install" to $prefix/share/doc/hwloc/ and your
|
|
|
|
|
systems default man page tree (under $prefix, of course).
|
|
|
|
|
NOTE: If you are building the documentation from a Git clone, you will need to
|
|
|
|
|
have Doxygen and pdflatex installed -- the documentation will be built during
|
|
|
|
|
the normal "make" process. The documentation is installed during "make install"
|
|
|
|
|
to $prefix/share/doc/hwloc/ and your systems default man page tree (under
|
|
|
|
|
$prefix, of course).
|
|
|
|
|
|
|
|
|
|
Portability
|
|
|
|
|
|
|
|
|
|
As shown in CLI Examples, hwloc can obtain information on a wide
|
|
|
|
|
variety of hardware topologies. However, some platforms and/or
|
|
|
|
|
operating system versions will only report a subset of this
|
|
|
|
|
information. For example, on an PPC64-based system with 32 cores (each
|
|
|
|
|
with 2 hardware threads) running a default 2.6.18-based kernel from
|
|
|
|
|
RHEL 5.4, hwloc is only able to glean information about NUMA nodes and
|
|
|
|
|
processor units (PUs). No information about caches, packages, or cores
|
|
|
|
|
is available.
|
|
|
|
|
As shown in CLI Examples, hwloc can obtain information on a wide variety of
|
|
|
|
|
hardware topologies. However, some platforms and/or operating system versions
|
|
|
|
|
will only report a subset of this information. For example, on an PPC64-based
|
|
|
|
|
system with 32 cores (each with 2 hardware threads) running a default
|
|
|
|
|
2.6.18-based kernel from RHEL 5.4, hwloc is only able to glean information
|
|
|
|
|
about NUMA nodes and processor units (PUs). No information about caches,
|
|
|
|
|
packages, or cores is available.
|
|
|
|
|
|
|
|
|
|
Similarly, Operating System have varying support for CPU and memory
|
|
|
|
|
binding, e.g. while some Operating Systems provide interfaces for all
|
|
|
|
|
kinds of CPU and memory bindings, some others provide only interfaces
|
|
|
|
|
for a limited number of kinds of CPU and memory binding, and some do
|
|
|
|
|
not provide any binding interface at all. Hwloc's binding functions
|
|
|
|
|
would then simply return the ENOSYS error (Function not implemented),
|
|
|
|
|
meaning that the underlying Operating System does not provide any
|
|
|
|
|
interface for them. CPU binding and Memory binding provide more
|
|
|
|
|
information on which hwloc binding functions should be preferred
|
|
|
|
|
because interfaces for them are usually available on the supported
|
|
|
|
|
Operating Systems.
|
|
|
|
|
Similarly, Operating System have varying support for CPU and memory binding,
|
|
|
|
|
e.g. while some Operating Systems provide interfaces for all kinds of CPU and
|
|
|
|
|
memory bindings, some others provide only interfaces for a limited number of
|
|
|
|
|
kinds of CPU and memory binding, and some do not provide any binding interface
|
|
|
|
|
at all. Hwloc's binding functions would then simply return the ENOSYS error
|
|
|
|
|
(Function not implemented), meaning that the underlying Operating System does
|
|
|
|
|
not provide any interface for them. CPU binding and Memory binding provide more
|
|
|
|
|
information on which hwloc binding functions should be preferred because
|
|
|
|
|
interfaces for them are usually available on the supported Operating Systems.
|
|
|
|
|
|
|
|
|
|
Here's the graphical output from lstopo on this platform when
|
|
|
|
|
Simultaneous Multi-Threading (SMT) is enabled:
|
|
|
|
|
Here's the graphical output from lstopo on this platform when Simultaneous
|
|
|
|
|
Multi-Threading (SMT) is enabled:
|
|
|
|
|
|
|
|
|
|
ppc64-with-smt.png
|
|
|
|
|
ppc64-with-smt.png
|
|
|
|
|
|
|
|
|
|
And here's the graphical output from lstopo on this platform when SMT
|
|
|
|
|
is disabled:
|
|
|
|
|
And here's the graphical output from lstopo on this platform when SMT is
|
|
|
|
|
disabled:
|
|
|
|
|
|
|
|
|
|
ppc64-without-smt.png
|
|
|
|
|
ppc64-without-smt.png
|
|
|
|
|
|
|
|
|
|
Notice that hwloc only sees half the PUs when SMT is disabled. PU #15,
|
|
|
|
|
for example, seems to change location from NUMA node #0 to #1. In
|
|
|
|
|
reality, no PUs "moved" -- they were simply re-numbered when hwloc only
|
|
|
|
|
saw half as many. Hence, PU #15 in the SMT-disabled picture probably
|
|
|
|
|
corresponds to PU #30 in the SMT-enabled picture.
|
|
|
|
|
Notice that hwloc only sees half the PUs when SMT is disabled. PU #15, for
|
|
|
|
|
example, seems to change location from NUMA node #0 to #1. In reality, no PUs
|
|
|
|
|
"moved" -- they were simply re-numbered when hwloc only saw half as many.
|
|
|
|
|
Hence, PU #15 in the SMT-disabled picture probably corresponds to PU #30 in the
|
|
|
|
|
SMT-enabled picture.
|
|
|
|
|
|
|
|
|
|
This same "PUs have disappeared" effect can be seen on other platforms
|
|
|
|
|
-- even platforms / OSs that provide much more information than the
|
|
|
|
|
above PPC64 system. This is an unfortunate side-effect of how operating
|
|
|
|
|
systems report information to hwloc.
|
|
|
|
|
This same "PUs have disappeared" effect can be seen on other platforms -- even
|
|
|
|
|
platforms / OSs that provide much more information than the above PPC64 system.
|
|
|
|
|
This is an unfortunate side-effect of how operating systems report information
|
|
|
|
|
to hwloc.
|
|
|
|
|
|
|
|
|
|
Note that upgrading the Linux kernel on the same PPC64 system mentioned
|
|
|
|
|
above to 2.6.34, hwloc is able to discover all the topology
|
|
|
|
|
information. The following picture shows the entire topology layout
|
|
|
|
|
when SMT is enabled:
|
|
|
|
|
Note that upgrading the Linux kernel on the same PPC64 system mentioned above
|
|
|
|
|
to 2.6.34, hwloc is able to discover all the topology information. The
|
|
|
|
|
following picture shows the entire topology layout when SMT is enabled:
|
|
|
|
|
|
|
|
|
|
ppc64-full-with-smt.png
|
|
|
|
|
ppc64-full-with-smt.png
|
|
|
|
|
|
|
|
|
|
Developers using the hwloc API or XML output for portable applications
|
|
|
|
|
should therefore be extremely careful to not make any assumptions about
|
|
|
|
|
the structure of data that is returned. For example, per the above
|
|
|
|
|
reported PPC topology, it is not safe to assume that PUs will always be
|
|
|
|
|
descendants of cores.
|
|
|
|
|
Developers using the hwloc API or XML output for portable applications should
|
|
|
|
|
therefore be extremely careful to not make any assumptions about the structure
|
|
|
|
|
of data that is returned. For example, per the above reported PPC topology, it
|
|
|
|
|
is not safe to assume that PUs will always be descendants of cores.
|
|
|
|
|
|
|
|
|
|
Additionally, future hardware may insert new topology elements that are
|
|
|
|
|
not available in this version of hwloc. Long-lived applications that
|
|
|
|
|
are meant to span multiple different hardware platforms should also be
|
|
|
|
|
careful about making structure assumptions. For example, there may
|
|
|
|
|
someday be an element "lower" than a PU, or perhaps a new element may
|
|
|
|
|
exist between a core and a PU.
|
|
|
|
|
Additionally, future hardware may insert new topology elements that are not
|
|
|
|
|
available in this version of hwloc. Long-lived applications that are meant to
|
|
|
|
|
span multiple different hardware platforms should also be careful about making
|
|
|
|
|
structure assumptions. For example, there may someday be an element "lower"
|
|
|
|
|
than a PU, or perhaps a new element may exist between a core and a PU.
|
|
|
|
|
|
|
|
|
|
API Example
|
|
|
|
|
|
|
|
|
|
The following small C example (named ``hwloc-hello.c'') prints the
|
|
|
|
|
topology of the machine and bring the process to the first logical
|
|
|
|
|
processor of the second core of the machine. More examples are
|
|
|
|
|
available in the doc/examples/ directory of the source tree.
|
|
|
|
|
The following small C example (named ``hwloc-hello.c'') prints the topology of
|
|
|
|
|
the machine and bring the process to the first logical processor of the second
|
|
|
|
|
core of the machine. More examples are available in the doc/examples/ directory
|
|
|
|
|
of the source tree.
|
|
|
|
|
|
|
|
|
|
/* Example hwloc API program.
|
|
|
|
|
*
|
|
|
|
|
* See other examples under doc/examples/ in the source tree
|
|
|
|
|
* for more details.
|
|
|
|
|
*
|
|
|
|
|
* Copyright (c) 2009-2015 Inria. All rights reserved.
|
|
|
|
|
* Copyright (c) 2009-2011 Universit?eacute; Bordeaux
|
|
|
|
|
* Copyright (c) 2009-2010 Cisco Systems, Inc. All rights reserved.
|
|
|
|
|
* See COPYING in top-level directory.
|
|
|
|
|
*
|
|
|
|
|
* hwloc-hello.c
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <hwloc.h>
|
|
|
|
|
#include <errno.h>
|
|
|
|
|
#include <stdio.h>
|
|
|
|
|
#include <string.h>
|
|
|
|
|
|
|
|
|
|
static void print_children(hwloc_topology_t topology, hwloc_obj_t obj,
|
|
|
|
|
int depth)
|
|
|
|
|
{
|
|
|
|
|
char type[32], attr[1024];
|
|
|
|
|
unsigned i;
|
|
|
|
|
|
|
|
|
|
hwloc_obj_type_snprintf(type, sizeof(type), obj, 0);
|
|
|
|
|
printf("%*s%s", 2*depth, "", type);
|
|
|
|
|
if (obj->os_index != (unsigned) -1)
|
|
|
|
|
printf("#%u", obj->os_index);
|
|
|
|
|
hwloc_obj_attr_snprintf(attr, sizeof(attr), obj, " ", 0);
|
|
|
|
|
if (*attr)
|
|
|
|
|
printf("(%s)", attr);
|
|
|
|
|
printf("\n");
|
|
|
|
|
for (i = 0; i < obj->arity; i++) {
|
|
|
|
|
print_children(topology, obj->children[i], depth + 1);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int main(void)
|
|
|
|
|
{
|
|
|
|
|
int depth;
|
|
|
|
|
unsigned i, n;
|
|
|
|
|
unsigned long size;
|
|
|
|
|
int levels;
|
|
|
|
|
char string[128];
|
|
|
|
|
int topodepth;
|
|
|
|
|
hwloc_topology_t topology;
|
|
|
|
|
hwloc_cpuset_t cpuset;
|
|
|
|
|
hwloc_obj_t obj;
|
|
|
|
|
|
|
|
|
|
/* Allocate and initialize topology object. */
|
|
|
|
|
hwloc_topology_init(&topology);
|
|
|
|
|
|
|
|
|
|
/* ... Optionally, put detection configuration here to ignore
|
|
|
|
|
some objects types, define a synthetic topology, etc....
|
|
|
|
|
|
|
|
|
|
The default is to detect all the objects of the machine that
|
|
|
|
|
the caller is allowed to access. See Configure Topology
|
|
|
|
|
Detection. */
|
|
|
|
|
|
|
|
|
|
/* Perform the topology detection. */
|
|
|
|
|
hwloc_topology_load(topology);
|
|
|
|
|
|
|
|
|
|
/* Optionally, get some additional topology information
|
|
|
|
|
in case we need the topology depth later. */
|
|
|
|
|
topodepth = hwloc_topology_get_depth(topology);
|
|
|
|
|
|
|
|
|
|
/*****************************************************************
|
|
|
|
|
* First example:
|
|
|
|
|
* Walk the topology with an array style, from level 0 (always
|
|
|
|
|
* the system level) to the lowest level (always the proc level).
|
|
|
|
|
*****************************************************************/
|
|
|
|
|
for (depth = 0; depth < topodepth; depth++) {
|
|
|
|
|
printf("*** Objects at level %d\n", depth);
|
|
|
|
|
for (i = 0; i < hwloc_get_nbobjs_by_depth(topology, depth);
|
|
|
|
|
i++) {
|
|
|
|
|
hwloc_obj_type_snprintf(string, sizeof(string),
|
|
|
|
|
hwloc_get_obj_by_depth
|
|
|
|
|
(topology, depth, i), 0);
|
|
|
|
|
printf("Index %u: %s\n", i, string);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*****************************************************************
|
|
|
|
|
* Second example:
|
|
|
|
|
* Walk the topology with a tree style.
|
|
|
|
|
*****************************************************************/
|
|
|
|
|
printf("*** Printing overall tree\n");
|
|
|
|
|
print_children(topology, hwloc_get_root_obj(topology), 0);
|
|
|
|
|
|
|
|
|
|
/*****************************************************************
|
|
|
|
|
* Third example:
|
|
|
|
|
* Print the number of packages.
|
|
|
|
|
*****************************************************************/
|
|
|
|
|
depth = hwloc_get_type_depth(topology, HWLOC_OBJ_PACKAGE);
|
|
|
|
|
if (depth == HWLOC_TYPE_DEPTH_UNKNOWN) {
|
|
|
|
|
printf("*** The number of packages is unknown\n");
|
|
|
|
|
} else {
|
|
|
|
|
printf("*** %u package(s)\n",
|
|
|
|
|
hwloc_get_nbobjs_by_depth(topology, depth));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*****************************************************************
|
|
|
|
|
* Fourth example:
|
|
|
|
|
* Compute the amount of cache that the first logical processor
|
|
|
|
|
* has above it.
|
|
|
|
|
*****************************************************************/
|
|
|
|
|
levels = 0;
|
|
|
|
|
size = 0;
|
|
|
|
|
for (obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_PU, 0);
|
|
|
|
|
obj;
|
|
|
|
|
obj = obj->parent)
|
|
|
|
|
if (obj->type == HWLOC_OBJ_CACHE) {
|
|
|
|
|
levels++;
|
|
|
|
|
size += obj->attr->cache.size;
|
|
|
|
|
}
|
|
|
|
|
printf("*** Logical processor 0 has %d caches totaling %luKB\n",
|
|
|
|
|
levels, size / 1024);
|
|
|
|
|
|
|
|
|
|
/*****************************************************************
|
|
|
|
|
* Fifth example:
|
|
|
|
|
* Bind to only one thread of the last core of the machine.
|
|
|
|
|
*
|
|
|
|
|
* First find out where cores are, or else smaller sets of CPUs if
|
|
|
|
|
* the OS doesn't have the notion of a "core".
|
|
|
|
|
*****************************************************************/
|
|
|
|
|
depth = hwloc_get_type_or_below_depth(topology, HWLOC_OBJ_CORE);
|
|
|
|
|
|
|
|
|
|
/* Get last core. */
|
|
|
|
|
obj = hwloc_get_obj_by_depth(topology, depth,
|
|
|
|
|
hwloc_get_nbobjs_by_depth(topology, depth) - 1);
|
|
|
|
|
if (obj) {
|
|
|
|
|
/* Get a copy of its cpuset that we may modify. */
|
|
|
|
|
cpuset = hwloc_bitmap_dup(obj->cpuset);
|
|
|
|
|
|
|
|
|
|
/* Get only one logical processor (in case the core is
|
|
|
|
|
SMT/hyper-threaded). */
|
|
|
|
|
hwloc_bitmap_singlify(cpuset);
|
|
|
|
|
|
|
|
|
|
/* And try to bind ourself there. */
|
|
|
|
|
if (hwloc_set_cpubind(topology, cpuset, 0)) {
|
|
|
|
|
char *str;
|
|
|
|
|
int error = errno;
|
|
|
|
|
hwloc_bitmap_asprintf(&str, obj->cpuset);
|
|
|
|
|
printf("Couldn't bind to cpuset %s: %s\n", str, strerror(error));
|
|
|
|
|
free(str);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Free our cpuset copy */
|
|
|
|
|
hwloc_bitmap_free(cpuset);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*****************************************************************
|
|
|
|
|
* Sixth example:
|
|
|
|
|
* Allocate some memory on the last NUMA node, bind some existing
|
|
|
|
|
* memory to the last NUMA node.
|
|
|
|
|
*****************************************************************/
|
|
|
|
|
/* Get last node. */
|
|
|
|
|
n = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_NUMANODE);
|
|
|
|
|
if (n) {
|
|
|
|
|
void *m;
|
|
|
|
|
size = 1024*1024;
|
|
|
|
|
|
|
|
|
|
obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NUMANODE, n - 1);
|
|
|
|
|
m = hwloc_alloc_membind_nodeset(topology, size, obj->nodeset,
|
|
|
|
|
HWLOC_MEMBIND_BIND, 0);
|
|
|
|
|
hwloc_free(topology, m, size);
|
|
|
|
|
|
|
|
|
|
m = malloc(size);
|
|
|
|
|
hwloc_set_area_membind_nodeset(topology, m, size, obj->nodeset,
|
|
|
|
|
HWLOC_MEMBIND_BIND, 0);
|
|
|
|
|
free(m);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Destroy topology object. */
|
|
|
|
|
hwloc_topology_destroy(topology);
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
hwloc provides a pkg-config executable to obtain relevant compiler and linker
|
|
|
|
|
flags. For example, it can be used thusly to compile applications that utilize
|
|
|
|
|
the hwloc library (assuming GNU Make):
|
|
|
|
|
|
|
|
|
|
hwloc provides a pkg-config executable to obtain relevant compiler and
|
|
|
|
|
linker flags. For example, it can be used thusly to compile
|
|
|
|
|
applications that utilize the hwloc library (assuming GNU Make):
|
|
|
|
|
CFLAGS += $(pkg-config --cflags hwloc)
|
|
|
|
|
LDLIBS += $(pkg-config --libs hwloc)
|
|
|
|
|
cc hwloc-hello.c $(CFLAGS) -o hwloc-hello $(LDLIBS)
|
|
|
|
|
|
|
|
|
|
On a machine with 4GB of RAM and 2 processor packages -- each package
|
|
|
|
|
of which has two processing cores -- the output from running
|
|
|
|
|
hwloc-hello could be something like the following:
|
|
|
|
|
On a machine with 4GB of RAM and 2 processor packages -- each package of which
|
|
|
|
|
has two processing cores -- the output from running hwloc-hello could be
|
|
|
|
|
something like the following:
|
|
|
|
|
|
|
|
|
|
shell$ ./hwloc-hello
|
|
|
|
|
*** Objects at level 0
|
|
|
|
|
Index 0: Machine(3938MB)
|
|
|
|
|
@ -383,45 +555,43 @@ shell$
|
|
|
|
|
|
|
|
|
|
Questions and Bugs
|
|
|
|
|
|
|
|
|
|
Questions should be sent to the devel mailing list
|
|
|
|
|
(http://www.open-mpi.org/community/lists/hwloc.php). Bug reports should
|
|
|
|
|
be reported in the tracker (https://git.open-mpi.org/trac/hwloc/).
|
|
|
|
|
Questions should be sent to the devel mailing list (http://www.open-mpi.org/
|
|
|
|
|
community/lists/hwloc.php). Bug reports should be reported in the tracker (
|
|
|
|
|
https://git.open-mpi.org/trac/hwloc/).
|
|
|
|
|
|
|
|
|
|
If hwloc discovers an incorrect topology for your machine, the very
|
|
|
|
|
first thing you should check is to ensure that you have the most recent
|
|
|
|
|
updates installed for your operating system. Indeed, most of hwloc
|
|
|
|
|
topology discovery relies on hardware information retrieved through the
|
|
|
|
|
operation system (e.g., via the /sys virtual filesystem of the Linux
|
|
|
|
|
kernel). If upgrading your OS or Linux kernel does not solve your
|
|
|
|
|
problem, you may also want to ensure that you are running the most
|
|
|
|
|
recent version of the BIOS for your machine.
|
|
|
|
|
If hwloc discovers an incorrect topology for your machine, the very first thing
|
|
|
|
|
you should check is to ensure that you have the most recent updates installed
|
|
|
|
|
for your operating system. Indeed, most of hwloc topology discovery relies on
|
|
|
|
|
hardware information retrieved through the operation system (e.g., via the /sys
|
|
|
|
|
virtual filesystem of the Linux kernel). If upgrading your OS or Linux kernel
|
|
|
|
|
does not solve your problem, you may also want to ensure that you are running
|
|
|
|
|
the most recent version of the BIOS for your machine.
|
|
|
|
|
|
|
|
|
|
If those things fail, contact us on the mailing list for additional
|
|
|
|
|
help. Please attach the output of lstopo after having given the
|
|
|
|
|
--enable-debug option to ./configure and rebuilt completely, to get
|
|
|
|
|
debugging output. Also attach the /proc + /sys tarball generated by the
|
|
|
|
|
installed script hwloc-gather-topology when submitting problems about
|
|
|
|
|
Linux, or send the output of kstat cpu_info in the Solaris case, or the
|
|
|
|
|
output of sysctl hw in the Darwin or BSD cases.
|
|
|
|
|
If those things fail, contact us on the mailing list for additional help.
|
|
|
|
|
Please attach the output of lstopo after having given the --enable-debug option
|
|
|
|
|
to ./configure and rebuilt completely, to get debugging output. Also attach the
|
|
|
|
|
/proc + /sys tarball generated by the installed script hwloc-gather-topology
|
|
|
|
|
when submitting problems about Linux, or send the output of kstat cpu_info in
|
|
|
|
|
the Solaris case, or the output of sysctl hw in the Darwin or BSD cases.
|
|
|
|
|
|
|
|
|
|
History / Credits
|
|
|
|
|
|
|
|
|
|
hwloc is the evolution and merger of the libtopology
|
|
|
|
|
(http://runtime.bordeaux.inria.fr/libtopology/) project and the
|
|
|
|
|
Portable Linux Processor Affinity (PLPA)
|
|
|
|
|
(http://www.open-mpi.org/projects/plpa/) project. Because of functional
|
|
|
|
|
and ideological overlap, these two code bases and ideas were merged and
|
|
|
|
|
released under the name "hwloc" as an Open MPI sub-project.
|
|
|
|
|
hwloc is the evolution and merger of the libtopology (http://
|
|
|
|
|
runtime.bordeaux.inria.fr/libtopology/) project and the Portable Linux
|
|
|
|
|
Processor Affinity (PLPA) (http://www.open-mpi.org/projects/plpa/) project.
|
|
|
|
|
Because of functional and ideological overlap, these two code bases and ideas
|
|
|
|
|
were merged and released under the name "hwloc" as an Open MPI sub-project.
|
|
|
|
|
|
|
|
|
|
libtopology was initially developed by the inria Runtime Team-Project
|
|
|
|
|
(http://runtime.bordeaux.inria.fr/) (headed by Raymond Namyst
|
|
|
|
|
(http://dept-info.labri.fr/~namyst/). PLPA was initially developed by
|
|
|
|
|
the Open MPI development team as a sub-project. Both are now deprecated
|
|
|
|
|
in favor of hwloc, which is distributed as an Open MPI sub-project.
|
|
|
|
|
libtopology was initially developed by the inria Runtime Team-Project (http://
|
|
|
|
|
runtime.bordeaux.inria.fr/) (headed by Raymond Namyst (http://
|
|
|
|
|
dept-info.labri.fr/~namyst/). PLPA was initially developed by the Open MPI
|
|
|
|
|
development team as a sub-project. Both are now deprecated in favor of hwloc,
|
|
|
|
|
which is distributed as an Open MPI sub-project.
|
|
|
|
|
|
|
|
|
|
Further Reading
|
|
|
|
|
|
|
|
|
|
The documentation chapters include
|
|
|
|
|
|
|
|
|
|
* Terms and Definitions
|
|
|
|
|
* Command-Line Tools
|
|
|
|
|
* Environment Variables
|
|
|
|
|
@ -439,8 +609,4 @@ The documentation chapters include
|
|
|
|
|
* Frequently Asked Questions
|
|
|
|
|
|
|
|
|
|
Make sure to have had a look at those too!
|
|
|
|
|
__________________________________________________________________
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Generated on 5 Jun 2015 for Hardware Locality (hwloc) by doxygen
|
|
|
|
|
1.6.1
|
|
|
|
|
|
Ralph Castain