This commit is the first of several steps in a paffinity makeover
extravaganza.
= Short version =
This commit does several things, but the short version is that it
re-orients the error message creation of the ODLS default module to
generate error strings in the child process for errors that occur
after the fork but before the exec (such errors are ''usually''
related to paffinity). A show_help string is rendered in the child
and then IPC'ed up to the parent, who displays the string through
normal ORTE show_help aggregation mechanisms. We also broke up the
ginormous paffinity-setting logic into a few separate functions, both
to help us understand the code, and hopefully to ease future
maintenance.
The logic for the ODLS default binding should not have changed -- this
is mainly a code reshuffle and improvement on error reporting.
= Rationale =
The reasoning for this commit is complex. As mentioned above, it's
the first step in some paffinity cleanup. Here's the line of dominoes
that must fall (in this order):
1. Add hwloc paffinity component (already done).
1. While testing hwloc, we discovered that the error reporting from
the ODLS default module was abysmal. So we fixed it.
1. Further, we reorganized the code in the odsl_default_module.c a bit
to help our understanding of it.
1. We also discovered a few bugs in the original ODLS default module
logic that existed before this code shuffle; separate tickets
will be filed to fix them.
1. Next up will be some improvements to paffinity / odls default to
make the act of binding to a core ensure to bind to ''all''
hardware threads contained in that core (similar for sockets:
binding to a socket will bind to ''all'' hardware threads in that
socket).
1. Next will be improvements to paffinity to expose binding to
hardware threads through the paffinity framework API.
1. Finally, we'll expose these binding controls to the user (e.g.,
through mpirun command line arguments, MCA parameters, etc.).
This commit represents the first few bullets; the last 4 bullets are
being worked on right now, but there is no definite timeline for
completion.
= Miscelaneous =
A few points worth mentioning:
* We have tested this new code a bunch; we're pretty sure it behaves
just like the trunk -- but with better / more precise error
reporting. More testing is needed on a wider array of platforms,
however.
* A big comment at the top of odls_default_module.c explains the
(new) general scheme for the error reporting.
* The error reporting in the parent process is now really dumb;
almost all the intelligence about creating error messages is in the
child.
* The show_help file was renamed to be more consistent with other
help files (help-odls-default.txt -> help-orte-odls-default.txt)
* Removed the use of sched_yield() because of recent changes in the
Linux 2.6.3x kernels. We already had an #else clause for
select()'ing for 1us if we didn't have sched_yield() -- that is now
the only code path. This is not a performance-critical section of
the code, so this shouldn't be controversial.
* Replaced the macro-based error reporting with function-based
reporting. It's a bit more bulky, but it helped us understand the
code and saved us multiple times with compile-time parameter
checking, etc.
* Cleaned up the use of several show_help messages to ensure that
they mapped to real messages in help*.txt files.
This commit was SVN r23652.
2010-08-24 19:38:29 +00:00
|
|
|
# -*- text -*-
|
|
|
|
|
#
|
|
|
|
|
# Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
|
|
|
|
# University Research and Technology
|
|
|
|
|
# Corporation. All rights reserved.
|
|
|
|
|
# Copyright (c) 2004-2005 The University of Tennessee and The University
|
|
|
|
|
# of Tennessee Research Foundation. All rights
|
|
|
|
|
# reserved.
|
|
|
|
|
# Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
|
|
|
# University of Stuttgart. All rights reserved.
|
|
|
|
|
# Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
|
# All rights reserved.
|
|
|
|
|
# Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved.
|
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 03:40:11 +00:00
|
|
|
# Copyright (c) 2010-2011 Cisco Systems, Inc. All rights reserved.
|
This commit is the first of several steps in a paffinity makeover
extravaganza.
= Short version =
This commit does several things, but the short version is that it
re-orients the error message creation of the ODLS default module to
generate error strings in the child process for errors that occur
after the fork but before the exec (such errors are ''usually''
related to paffinity). A show_help string is rendered in the child
and then IPC'ed up to the parent, who displays the string through
normal ORTE show_help aggregation mechanisms. We also broke up the
ginormous paffinity-setting logic into a few separate functions, both
to help us understand the code, and hopefully to ease future
maintenance.
The logic for the ODLS default binding should not have changed -- this
is mainly a code reshuffle and improvement on error reporting.
= Rationale =
The reasoning for this commit is complex. As mentioned above, it's
the first step in some paffinity cleanup. Here's the line of dominoes
that must fall (in this order):
1. Add hwloc paffinity component (already done).
1. While testing hwloc, we discovered that the error reporting from
the ODLS default module was abysmal. So we fixed it.
1. Further, we reorganized the code in the odsl_default_module.c a bit
to help our understanding of it.
1. We also discovered a few bugs in the original ODLS default module
logic that existed before this code shuffle; separate tickets
will be filed to fix them.
1. Next up will be some improvements to paffinity / odls default to
make the act of binding to a core ensure to bind to ''all''
hardware threads contained in that core (similar for sockets:
binding to a socket will bind to ''all'' hardware threads in that
socket).
1. Next will be improvements to paffinity to expose binding to
hardware threads through the paffinity framework API.
1. Finally, we'll expose these binding controls to the user (e.g.,
through mpirun command line arguments, MCA parameters, etc.).
This commit represents the first few bullets; the last 4 bullets are
being worked on right now, but there is no definite timeline for
completion.
= Miscelaneous =
A few points worth mentioning:
* We have tested this new code a bunch; we're pretty sure it behaves
just like the trunk -- but with better / more precise error
reporting. More testing is needed on a wider array of platforms,
however.
* A big comment at the top of odls_default_module.c explains the
(new) general scheme for the error reporting.
* The error reporting in the parent process is now really dumb;
almost all the intelligence about creating error messages is in the
child.
* The show_help file was renamed to be more consistent with other
help files (help-odls-default.txt -> help-orte-odls-default.txt)
* Removed the use of sched_yield() because of recent changes in the
Linux 2.6.3x kernels. We already had an #else clause for
select()'ing for 1us if we didn't have sched_yield() -- that is now
the only code path. This is not a performance-critical section of
the code, so this shouldn't be controversial.
* Replaced the macro-based error reporting with function-based
reporting. It's a bit more bulky, but it helped us understand the
code and saved us multiple times with compile-time parameter
checking, etc.
* Cleaned up the use of several show_help messages to ensure that
they mapped to real messages in help*.txt files.
This commit was SVN r23652.
2010-08-24 19:38:29 +00:00
|
|
|
# $COPYRIGHT$
|
|
|
|
|
#
|
|
|
|
|
# Additional copyrights may follow
|
|
|
|
|
#
|
|
|
|
|
# $HEADER$
|
|
|
|
|
#
|
|
|
|
|
# This is a US/English help file.
|
|
|
|
|
#
|
|
|
|
|
[execve error]
|
|
|
|
|
Open MPI tried to fork a new process via the "execve" system call but
|
2013-02-27 01:36:21 +00:00
|
|
|
failed. Open MPI checks many things before attempting to launch a
|
|
|
|
|
child process, but nothing is perfect. This error may be indicative
|
|
|
|
|
of another problem on the target host, or even something as silly as
|
|
|
|
|
having specified a directory for your application. Your job will now
|
This commit is the first of several steps in a paffinity makeover
extravaganza.
= Short version =
This commit does several things, but the short version is that it
re-orients the error message creation of the ODLS default module to
generate error strings in the child process for errors that occur
after the fork but before the exec (such errors are ''usually''
related to paffinity). A show_help string is rendered in the child
and then IPC'ed up to the parent, who displays the string through
normal ORTE show_help aggregation mechanisms. We also broke up the
ginormous paffinity-setting logic into a few separate functions, both
to help us understand the code, and hopefully to ease future
maintenance.
The logic for the ODLS default binding should not have changed -- this
is mainly a code reshuffle and improvement on error reporting.
= Rationale =
The reasoning for this commit is complex. As mentioned above, it's
the first step in some paffinity cleanup. Here's the line of dominoes
that must fall (in this order):
1. Add hwloc paffinity component (already done).
1. While testing hwloc, we discovered that the error reporting from
the ODLS default module was abysmal. So we fixed it.
1. Further, we reorganized the code in the odsl_default_module.c a bit
to help our understanding of it.
1. We also discovered a few bugs in the original ODLS default module
logic that existed before this code shuffle; separate tickets
will be filed to fix them.
1. Next up will be some improvements to paffinity / odls default to
make the act of binding to a core ensure to bind to ''all''
hardware threads contained in that core (similar for sockets:
binding to a socket will bind to ''all'' hardware threads in that
socket).
1. Next will be improvements to paffinity to expose binding to
hardware threads through the paffinity framework API.
1. Finally, we'll expose these binding controls to the user (e.g.,
through mpirun command line arguments, MCA parameters, etc.).
This commit represents the first few bullets; the last 4 bullets are
being worked on right now, but there is no definite timeline for
completion.
= Miscelaneous =
A few points worth mentioning:
* We have tested this new code a bunch; we're pretty sure it behaves
just like the trunk -- but with better / more precise error
reporting. More testing is needed on a wider array of platforms,
however.
* A big comment at the top of odls_default_module.c explains the
(new) general scheme for the error reporting.
* The error reporting in the parent process is now really dumb;
almost all the intelligence about creating error messages is in the
child.
* The show_help file was renamed to be more consistent with other
help files (help-odls-default.txt -> help-orte-odls-default.txt)
* Removed the use of sched_yield() because of recent changes in the
Linux 2.6.3x kernels. We already had an #else clause for
select()'ing for 1us if we didn't have sched_yield() -- that is now
the only code path. This is not a performance-critical section of
the code, so this shouldn't be controversial.
* Replaced the macro-based error reporting with function-based
reporting. It's a bit more bulky, but it helped us understand the
code and saved us multiple times with compile-time parameter
checking, etc.
* Cleaned up the use of several show_help messages to ensure that
they mapped to real messages in help*.txt files.
This commit was SVN r23652.
2010-08-24 19:38:29 +00:00
|
|
|
abort.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
2013-02-27 01:36:21 +00:00
|
|
|
Error: %s
|
This commit is the first of several steps in a paffinity makeover
extravaganza.
= Short version =
This commit does several things, but the short version is that it
re-orients the error message creation of the ODLS default module to
generate error strings in the child process for errors that occur
after the fork but before the exec (such errors are ''usually''
related to paffinity). A show_help string is rendered in the child
and then IPC'ed up to the parent, who displays the string through
normal ORTE show_help aggregation mechanisms. We also broke up the
ginormous paffinity-setting logic into a few separate functions, both
to help us understand the code, and hopefully to ease future
maintenance.
The logic for the ODLS default binding should not have changed -- this
is mainly a code reshuffle and improvement on error reporting.
= Rationale =
The reasoning for this commit is complex. As mentioned above, it's
the first step in some paffinity cleanup. Here's the line of dominoes
that must fall (in this order):
1. Add hwloc paffinity component (already done).
1. While testing hwloc, we discovered that the error reporting from
the ODLS default module was abysmal. So we fixed it.
1. Further, we reorganized the code in the odsl_default_module.c a bit
to help our understanding of it.
1. We also discovered a few bugs in the original ODLS default module
logic that existed before this code shuffle; separate tickets
will be filed to fix them.
1. Next up will be some improvements to paffinity / odls default to
make the act of binding to a core ensure to bind to ''all''
hardware threads contained in that core (similar for sockets:
binding to a socket will bind to ''all'' hardware threads in that
socket).
1. Next will be improvements to paffinity to expose binding to
hardware threads through the paffinity framework API.
1. Finally, we'll expose these binding controls to the user (e.g.,
through mpirun command line arguments, MCA parameters, etc.).
This commit represents the first few bullets; the last 4 bullets are
being worked on right now, but there is no definite timeline for
completion.
= Miscelaneous =
A few points worth mentioning:
* We have tested this new code a bunch; we're pretty sure it behaves
just like the trunk -- but with better / more precise error
reporting. More testing is needed on a wider array of platforms,
however.
* A big comment at the top of odls_default_module.c explains the
(new) general scheme for the error reporting.
* The error reporting in the parent process is now really dumb;
almost all the intelligence about creating error messages is in the
child.
* The show_help file was renamed to be more consistent with other
help files (help-odls-default.txt -> help-orte-odls-default.txt)
* Removed the use of sched_yield() because of recent changes in the
Linux 2.6.3x kernels. We already had an #else clause for
select()'ing for 1us if we didn't have sched_yield() -- that is now
the only code path. This is not a performance-critical section of
the code, so this shouldn't be controversial.
* Replaced the macro-based error reporting with function-based
reporting. It's a bit more bulky, but it helped us understand the
code and saved us multiple times with compile-time parameter
checking, etc.
* Cleaned up the use of several show_help messages to ensure that
they mapped to real messages in help*.txt files.
This commit was SVN r23652.
2010-08-24 19:38:29 +00:00
|
|
|
#
|
|
|
|
|
[binding not supported]
|
|
|
|
|
Open MPI tried to bind a new process, but process binding is not
|
|
|
|
|
supported on the host where it was launched. The process was killed
|
|
|
|
|
without launching the target application. Your job will now abort.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
#
|
|
|
|
|
[binding generic error]
|
|
|
|
|
Open MPI tried to bind a new process, but something went wrong. The
|
|
|
|
|
process was killed without launching the target application. Your job
|
|
|
|
|
will now abort.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
Error message: %s
|
|
|
|
|
Location: %s:%d
|
|
|
|
|
#
|
|
|
|
|
[bound to everything]
|
|
|
|
|
Open MPI tried to bind a new process to a specific set of processors,
|
|
|
|
|
but ended up binding it to *all* processors. This means that the new
|
|
|
|
|
process is effectively unbound.
|
|
|
|
|
|
|
|
|
|
This is only a warning -- your job will continue. You can suppress
|
|
|
|
|
this warning in the future by setting the odls_warn_if_not_bound MCA
|
|
|
|
|
parameter to 0.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
Location: %s:%d
|
|
|
|
|
#
|
|
|
|
|
[slot list and paffinity_alone]
|
|
|
|
|
Open MPI detected that both a slot list was specified and the MCA
|
|
|
|
|
parameter "paffinity_alone" was set to true. Only one of these can be
|
|
|
|
|
used at a time. Your job will now abort.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
#
|
|
|
|
|
[iof setup failed]
|
|
|
|
|
Open MPI tried to launch a child process but the "IOF child setup"
|
|
|
|
|
failed. This should not happen. Your job will now abort.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
#
|
|
|
|
|
[not bound]
|
|
|
|
|
WARNING: Open MPI tried to bind a process but failed. This is a
|
|
|
|
|
warning only; your job will continue.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
Error message: %s
|
|
|
|
|
Location: %s:%d
|
|
|
|
|
#
|
|
|
|
|
[syscall fail]
|
|
|
|
|
A system call failed that should not have. In this particular case,
|
|
|
|
|
a warning or error message was not displayed that should have been.
|
|
|
|
|
Your job may behave unpredictably after this, or abort.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
Function: %s
|
|
|
|
|
Location: %s:%d
|
At long last, the fabled revision to the affinity system has arrived. A more detailed explanation of how this all works will be presented here:
https://svn.open-mpi.org/trac/ompi/wiki/ProcessPlacement
The wiki page is incomplete at the moment, but I hope to complete it over the next few days. I will provide updates on the devel list. As the wiki page states, the default and most commonly used options remain unchanged (except as noted below). New, esoteric and complex options have been added, but unless you are a true masochist, you are unlikely to use many of them beyond perhaps an initial curiosity-motivated experimentation.
In a nutshell, this commit revamps the map/rank/bind procedure to take into account topology info on the compute nodes. I have, for the most part, preserved the default behaviors, with three notable exceptions:
1. I have at long last bowed my head in submission to the system admin's of managed clusters. For years, they have complained about our default of allowing users to oversubscribe nodes - i.e., to run more processes on a node than allocated slots. Accordingly, I have modified the default behavior: if you are running off of hostfile/dash-host allocated nodes, then the default is to allow oversubscription. If you are running off of RM-allocated nodes, then the default is to NOT allow oversubscription. Flags to override these behaviors are provided, so this only affects the default behavior.
2. both cpus/rank and stride have been removed. The latter was demanded by those who didn't understand the purpose behind it - and I agreed as the users who requested it are no longer using it. The former was removed temporarily pending implementation.
3. vm launch is now the sole method for starting OMPI. It was just too darned hard to maintain multiple launch procedures - maybe someday, provided someone can demonstrate a reason to do so.
As Jeff stated, it is impossible to fully test a change of this size. I have tested it on Linux and Mac, covering all the default and simple options, singletons, and comm_spawn. That said, I'm sure others will find problems, so I'll be watching MTT results until this stabilizes.
This commit was SVN r25476.
2011-11-15 03:40:11 +00:00
|
|
|
#
|
|
|
|
|
[memory not bound]
|
|
|
|
|
WARNING: Open MPI tried to bind a process but failed. This is a
|
|
|
|
|
warning only; your job will continue, though performance may
|
|
|
|
|
be degraded.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
Error message: %s
|
|
|
|
|
Location: %s:%d
|
|
|
|
|
|
|
|
|
|
#
|
|
|
|
|
[memory binding error]
|
|
|
|
|
Open MPI tried to bind memory for a new process but something went
|
|
|
|
|
wrong. The process was killed without launching the target
|
|
|
|
|
application. Your job will now abort.
|
|
|
|
|
|
|
|
|
|
Local host: %s
|
|
|
|
|
Application name: %s
|
|
|
|
|
Error message: %s
|
|
|
|
|
Location: %s:%d
|