2005-03-14 23:57:21 +03:00
|
|
|
/*
|
2007-03-29 04:50:56 +04:00
|
|
|
* Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
2005-11-05 22:57:48 +03:00
|
|
|
* University Research and Technology
|
|
|
|
* Corporation. All rights reserved.
|
2006-08-23 07:32:36 +04:00
|
|
|
* Copyright (c) 2004-2006 The University of Tennessee and The University
|
2005-11-05 22:57:48 +03:00
|
|
|
* of Tennessee Research Foundation. All rights
|
|
|
|
* reserved.
|
2005-03-14 23:57:21 +03:00
|
|
|
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
|
|
|
* University of Stuttgart. All rights reserved.
|
2005-03-24 15:43:37 +03:00
|
|
|
* Copyright (c) 2004-2005 The Regents of the University of California.
|
|
|
|
* All rights reserved.
|
2006-07-05 00:12:35 +04:00
|
|
|
* Copyright (c) 2006 Cisco Systems, Inc. All rights reserved.
|
2005-03-14 23:57:21 +03:00
|
|
|
* $COPYRIGHT$
|
|
|
|
*
|
|
|
|
* Additional copyrights may follow
|
|
|
|
*
|
|
|
|
* $HEADER$
|
|
|
|
*/
|
2006-07-05 00:12:35 +04:00
|
|
|
|
2005-03-14 23:57:21 +03:00
|
|
|
#include "orte_config.h"
|
2008-02-28 04:57:57 +03:00
|
|
|
#include "orte/constants.h"
|
|
|
|
#include "orte/types.h"
|
2006-07-10 18:10:21 +04:00
|
|
|
|
2005-03-14 23:57:21 +03:00
|
|
|
#include <errno.h>
|
2005-12-12 23:04:00 +03:00
|
|
|
#ifdef HAVE_UNISTD_H
|
2005-03-14 23:57:21 +03:00
|
|
|
#include <unistd.h>
|
2005-12-12 23:04:00 +03:00
|
|
|
#endif /* HAVE_UNISTD_H */
|
|
|
|
#ifdef HAVE_STRING_H
|
2005-03-14 23:57:21 +03:00
|
|
|
#include <string.h>
|
2005-12-12 23:04:00 +03:00
|
|
|
#endif /* HAVE_STRING_H */
|
2005-03-14 23:57:21 +03:00
|
|
|
|
2006-07-05 00:12:35 +04:00
|
|
|
#include "opal/mca/base/mca_base_param.h"
|
2005-07-04 03:31:27 +04:00
|
|
|
#include "opal/util/output.h"
|
2006-10-07 19:45:24 +04:00
|
|
|
#include "opal/util/trace.h"
|
2005-09-30 18:30:21 +04:00
|
|
|
#include "opal/util/show_help.h"
|
2005-10-08 02:24:52 +04:00
|
|
|
#include "opal/util/argv.h"
|
2006-07-10 18:10:21 +04:00
|
|
|
|
|
|
|
#include "orte/mca/errmgr/errmgr.h"
|
2006-09-15 01:29:51 +04:00
|
|
|
|
|
|
|
#include "orte/mca/rmaps/base/rmaps_private.h"
|
2006-10-07 23:50:12 +04:00
|
|
|
#include "orte/mca/rmaps/base/base.h"
|
2005-03-14 23:57:21 +03:00
|
|
|
#include "rmaps_rr.h"
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
* Local variable
|
2005-03-14 23:57:21 +03:00
|
|
|
*/
|
2005-07-03 20:22:16 +04:00
|
|
|
static opal_list_item_t *cur_node_item = NULL;
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
|
|
|
|
|
2007-07-14 19:14:07 +04:00
|
|
|
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
static int map_app_by_user_map(
|
2007-07-14 19:14:07 +04:00
|
|
|
orte_app_context_t* app,
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_job_t* jdata,
|
2007-07-14 19:14:07 +04:00
|
|
|
orte_vpid_t vpid_start,
|
|
|
|
opal_list_t* nodes,
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_list_t* procs)
|
2007-07-14 19:14:07 +04:00
|
|
|
{
|
2008-02-28 04:57:57 +03:00
|
|
|
#if 0
|
2007-07-14 19:14:07 +04:00
|
|
|
int rc = ORTE_SUCCESS;
|
|
|
|
orte_ras_proc_t *proc;
|
|
|
|
opal_list_item_t *proc_item, *node_item;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_node_t *node;
|
2007-07-14 19:14:07 +04:00
|
|
|
|
|
|
|
|
|
|
|
for (proc_item = opal_list_get_first(procs);
|
|
|
|
proc_item != opal_list_get_end(procs);
|
|
|
|
proc_item = opal_list_get_next(proc_item)) {
|
|
|
|
proc = (orte_ras_proc_t *)proc_item;
|
|
|
|
if(proc->rank >= vpid_start && proc->rank < (vpid_start + app->num_procs)){
|
|
|
|
for (node_item = opal_list_get_first(nodes);
|
|
|
|
node_item != opal_list_get_end(nodes);
|
|
|
|
node_item = opal_list_get_next(node_item)) {
|
2008-02-28 04:57:57 +03:00
|
|
|
node = (orte_node_t *)node_item;
|
|
|
|
if(0 == strcmp(node->name, proc->node_name)){
|
2007-07-14 19:14:07 +04:00
|
|
|
if (ORTE_SUCCESS != (rc = orte_rmaps_base_claim_slot(map, node, jobid, proc->rank, app->idx,
|
2008-02-28 04:57:57 +03:00
|
|
|
nodes, false))) {
|
2007-07-14 19:14:07 +04:00
|
|
|
/** if the code is ORTE_ERR_NODE_FULLY_USED, then we know this
|
|
|
|
* really isn't an error - we just need to break from the loop
|
|
|
|
* since the node is fully used up. For now, just don't report
|
|
|
|
* an error
|
|
|
|
*/
|
|
|
|
if (ORTE_ERR_NODE_FULLY_USED != rc) {
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
#endif
|
2007-07-14 19:14:07 +04:00
|
|
|
return ORTE_SUCCESS;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
/*
|
|
|
|
* Create a default mapping for the application, scheduling round
|
|
|
|
* robin by node.
|
|
|
|
*/
|
|
|
|
static int map_app_by_node(
|
2005-08-25 19:42:41 +04:00
|
|
|
orte_app_context_t* app,
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_job_t* jdata,
|
2005-03-14 23:57:21 +03:00
|
|
|
orte_vpid_t vpid_start,
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_list_t* nodes)
|
2005-03-14 23:57:21 +03:00
|
|
|
{
|
2005-10-08 21:03:12 +04:00
|
|
|
int rc = ORTE_SUCCESS;
|
2005-10-08 02:24:52 +04:00
|
|
|
opal_list_item_t *next;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_node_t *node;
|
|
|
|
orte_std_cntr_t num_alloc=0;
|
2005-03-14 23:57:21 +03:00
|
|
|
|
2006-10-07 19:45:24 +04:00
|
|
|
OPAL_TRACE(2);
|
|
|
|
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
/* This loop continues until all procs have been mapped or we run
|
2005-10-08 02:24:52 +04:00
|
|
|
out of resources. We determine that we have "run out of
|
2008-02-28 04:57:57 +03:00
|
|
|
resources" when all nodes have slots_max processes mapped to them,
|
2006-07-10 18:10:21 +04:00
|
|
|
thus there are no free slots for a process to be mapped, or we have
|
|
|
|
hit the soft limit on all nodes and are in a "no oversubscribe" state.
|
|
|
|
If we still have processes that haven't been mapped yet, then it's an
|
|
|
|
"out of resources" error.
|
|
|
|
|
|
|
|
In this scenario, we rely on the claim_slot function to handle the
|
|
|
|
oversubscribed case. The claim_slot function will leave a node on the
|
2008-02-28 04:57:57 +03:00
|
|
|
list until it either reaches slots_max OR reaches the
|
|
|
|
soft limit and the "no_oversubscribe" flag has been set - at which point,
|
2006-07-10 18:10:21 +04:00
|
|
|
the node will be removed to prevent any more processes from being mapped to
|
|
|
|
it. Since we are taking one slot from each node as we cycle through, the
|
|
|
|
list, oversubscription is automatically taken care of via this logic.
|
|
|
|
*/
|
|
|
|
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
while (num_alloc < app->num_procs) {
|
2005-10-08 02:24:52 +04:00
|
|
|
|
2006-07-10 18:10:21 +04:00
|
|
|
/** see if any nodes remain unused and available. We need to do this check
|
|
|
|
* each time since we may remove nodes from the list (as they become fully
|
|
|
|
* used) as we cycle through the loop */
|
|
|
|
if(0 >= opal_list_get_size(nodes) ) {
|
|
|
|
/* No more nodes to allocate :( */
|
|
|
|
opal_show_help("help-orte-rmaps-rr.txt", "orte-rmaps-rr:alloc-error",
|
|
|
|
true, app->num_procs, app->app);
|
2008-02-28 04:57:57 +03:00
|
|
|
return ORTE_ERR_SILENT;
|
2006-07-10 18:10:21 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Save the next node we can use before claiming slots, since
|
|
|
|
* we may need to prune the nodes list removing overused nodes.
|
|
|
|
* Wrap around to beginning if we are at the end of the list */
|
|
|
|
if (opal_list_get_end(nodes) == opal_list_get_next(cur_node_item)) {
|
|
|
|
next = opal_list_get_first(nodes);
|
2005-10-08 02:24:52 +04:00
|
|
|
}
|
|
|
|
else {
|
2006-07-10 18:10:21 +04:00
|
|
|
next = opal_list_get_next(cur_node_item);
|
2005-10-08 02:24:52 +04:00
|
|
|
}
|
2006-07-10 18:10:21 +04:00
|
|
|
|
|
|
|
/* Allocate a slot on this node */
|
2008-02-28 04:57:57 +03:00
|
|
|
node = (orte_node_t*) cur_node_item;
|
|
|
|
if (ORTE_SUCCESS != (rc = orte_rmaps_base_claim_slot(jdata, node, vpid_start + num_alloc, app->idx,
|
|
|
|
nodes, jdata->map->oversubscribe))) {
|
2006-10-20 03:33:51 +04:00
|
|
|
/** if the code is ORTE_ERR_NODE_FULLY_USED, then we know this
|
|
|
|
* really isn't an error - we just need to break from the loop
|
|
|
|
* since the node is fully used up. For now, just don't report
|
|
|
|
* an error
|
|
|
|
*/
|
|
|
|
if (ORTE_ERR_NODE_FULLY_USED != rc) {
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
return rc;
|
|
|
|
}
|
2005-10-08 02:24:52 +04:00
|
|
|
}
|
2005-06-09 20:00:41 +04:00
|
|
|
|
2006-07-10 18:10:21 +04:00
|
|
|
++num_alloc;
|
2005-03-14 23:57:21 +03:00
|
|
|
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
cur_node_item = next;
|
2005-10-08 02:24:52 +04:00
|
|
|
}
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
|
2006-07-10 18:10:21 +04:00
|
|
|
return ORTE_SUCCESS;
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a default mapping for the application, scheduling one round
|
|
|
|
* robin by slot.
|
|
|
|
*/
|
|
|
|
static int map_app_by_slot(
|
2005-08-25 19:42:41 +04:00
|
|
|
orte_app_context_t* app,
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_job_t* jdata,
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
orte_vpid_t vpid_start,
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_list_t* nodes)
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
{
|
2005-10-08 02:24:52 +04:00
|
|
|
int rc = ORTE_SUCCESS;
|
2006-08-15 23:54:10 +04:00
|
|
|
orte_std_cntr_t i, num_slots_to_take;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_node_t *node;
|
2006-07-10 18:10:21 +04:00
|
|
|
opal_list_item_t *next;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_std_cntr_t num_alloc=0;
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
|
2006-10-07 19:45:24 +04:00
|
|
|
OPAL_TRACE(2);
|
|
|
|
|
2005-10-08 02:24:52 +04:00
|
|
|
/* This loop continues until all procs have been mapped or we run
|
|
|
|
out of resources. We determine that we have "run out of
|
2008-02-28 04:57:57 +03:00
|
|
|
resources" when either all nodes have slots_max processes mapped to them,
|
2006-07-10 18:10:21 +04:00
|
|
|
(thus there are no free slots for a process to be mapped), OR all nodes
|
|
|
|
have reached their soft limit and the user directed us to "no oversubscribe".
|
|
|
|
If we still have processes that haven't been mapped yet, then it's an
|
2005-10-08 02:24:52 +04:00
|
|
|
"out of resources" error. */
|
2006-07-10 18:10:21 +04:00
|
|
|
|
2005-10-08 02:24:52 +04:00
|
|
|
while ( num_alloc < app->num_procs) {
|
2006-07-10 18:10:21 +04:00
|
|
|
|
|
|
|
/** see if any nodes remain unused and available. We need to do this check
|
|
|
|
* each time since we may remove nodes from the list (as they become fully
|
|
|
|
* used) as we cycle through the loop */
|
|
|
|
if(0 >= opal_list_get_size(nodes) ) {
|
2006-10-17 23:35:00 +04:00
|
|
|
/* Everything is at max usage! :( */
|
2006-07-10 18:10:21 +04:00
|
|
|
opal_show_help("help-orte-rmaps-rr.txt", "orte-rmaps-rr:alloc-error",
|
|
|
|
true, app->num_procs, app->app);
|
2008-02-28 04:57:57 +03:00
|
|
|
return ORTE_ERR_SILENT;
|
2006-07-10 18:10:21 +04:00
|
|
|
}
|
2005-10-08 02:24:52 +04:00
|
|
|
|
2006-07-10 18:10:21 +04:00
|
|
|
/* Save the next node we can use before claiming slots, since
|
|
|
|
* we may need to prune the nodes list removing overused nodes.
|
|
|
|
* Wrap around to beginning if we are at the end of the list */
|
|
|
|
if (opal_list_get_end(nodes) == opal_list_get_next(cur_node_item)) {
|
|
|
|
next = opal_list_get_first(nodes);
|
2005-10-08 02:24:52 +04:00
|
|
|
}
|
|
|
|
else {
|
2006-07-10 18:10:21 +04:00
|
|
|
next = opal_list_get_next(cur_node_item);
|
2005-10-08 02:24:52 +04:00
|
|
|
}
|
2006-07-10 18:10:21 +04:00
|
|
|
|
|
|
|
/** declare a shorter name for convenience in the code below */
|
2008-02-28 04:57:57 +03:00
|
|
|
node = (orte_node_t*) cur_node_item;
|
2006-07-10 18:10:21 +04:00
|
|
|
|
2005-10-08 02:24:52 +04:00
|
|
|
/* If we have available slots on this node, claim all of them
|
|
|
|
* If node_slots == 0, assume 1 slot for that node.
|
2006-07-10 18:10:21 +04:00
|
|
|
* JJH - is this assumption fully justified?
|
|
|
|
*
|
2006-10-20 22:24:14 +04:00
|
|
|
* If we are now oversubscribing the nodes, then we still take:
|
|
|
|
* (a) if the node has not been used yet, we take a full node_slots
|
|
|
|
* (b) if some of the slots are in-use, then we take the number of
|
|
|
|
* remaining slots before hitting the soft limit (node_slots)
|
|
|
|
* (c) if we are at or above the soft limit, we take a full node_slots
|
|
|
|
*
|
|
|
|
* Note: if node_slots is zero, then we always just take 1 slot
|
|
|
|
*
|
|
|
|
* We continue this process until either everything is done,
|
|
|
|
* or all nodes have hit their hard limit. This algorithm ensures we
|
|
|
|
* fully utilize each node before oversubscribing, and preserves the ratio
|
|
|
|
* of processes between the nodes thereafter (e.g., if one node has twice as
|
2006-07-10 18:10:21 +04:00
|
|
|
* many processes as another before oversubscribing, it will continue
|
|
|
|
* to do so after oversubscribing).
|
|
|
|
*/
|
2008-02-28 04:57:57 +03:00
|
|
|
if (0 == node->slots_inuse ||
|
|
|
|
node->slots_inuse >= node->slots) {
|
|
|
|
num_slots_to_take = (node->slots == 0) ? 1 : node->slots;
|
2006-10-20 22:24:14 +04:00
|
|
|
} else {
|
2008-02-28 04:57:57 +03:00
|
|
|
num_slots_to_take = node->slots - node->slots_inuse;
|
2006-10-20 22:24:14 +04:00
|
|
|
}
|
2006-07-10 18:10:21 +04:00
|
|
|
|
2007-01-18 20:15:19 +03:00
|
|
|
/* check if we are in npernode mode - if so, then set the num_slots_to_take
|
|
|
|
* to the num_per_node
|
|
|
|
*/
|
2008-02-28 04:57:57 +03:00
|
|
|
if (jdata->map->pernode) {
|
|
|
|
num_slots_to_take = jdata->map->npernode;
|
2007-01-18 20:15:19 +03:00
|
|
|
}
|
|
|
|
|
2006-07-10 18:10:21 +04:00
|
|
|
for( i = 0; i < num_slots_to_take; ++i) {
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (rc = orte_rmaps_base_claim_slot(jdata, node, vpid_start + num_alloc, app->idx,
|
|
|
|
nodes, jdata->map->oversubscribe))) {
|
2006-07-10 18:10:21 +04:00
|
|
|
/** if the code is ORTE_ERR_NODE_FULLY_USED, then we know this
|
|
|
|
* really isn't an error - we just need to break from the loop
|
|
|
|
* since the node is fully used up. For now, just don't report
|
|
|
|
* an error
|
|
|
|
*/
|
|
|
|
if (ORTE_ERR_NODE_FULLY_USED != rc) {
|
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
return rc;
|
2005-10-08 02:24:52 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-07-21 01:06:15 +04:00
|
|
|
/* Update the number of procs allocated */
|
2005-10-08 02:24:52 +04:00
|
|
|
++num_alloc;
|
2006-07-10 18:10:21 +04:00
|
|
|
|
|
|
|
/** if all the procs have been mapped OR we have fully used up this node, then
|
|
|
|
* break from the loop
|
|
|
|
*/
|
2006-10-19 22:57:29 +04:00
|
|
|
if(num_alloc == app->num_procs || ORTE_ERR_NODE_FULLY_USED == rc) {
|
2005-10-08 02:24:52 +04:00
|
|
|
break;
|
|
|
|
}
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
}
|
|
|
|
|
2006-10-20 22:05:16 +04:00
|
|
|
/* we move on to the next node in all cases EXCEPT if we came
|
|
|
|
* out of the loop without having taken a full bite AND the
|
|
|
|
* node is NOT max'd out
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
if (i < (num_slots_to_take-1) && ORTE_ERR_NODE_FULLY_USED != rc) {
|
|
|
|
continue;
|
2006-10-19 22:57:29 +04:00
|
|
|
}
|
2006-10-20 22:05:16 +04:00
|
|
|
cur_node_item = next;
|
2005-08-25 19:42:41 +04:00
|
|
|
}
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
|
2006-07-10 18:10:21 +04:00
|
|
|
return ORTE_SUCCESS;
|
2005-03-14 23:57:21 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
2006-07-10 18:10:21 +04:00
|
|
|
* Create a round-robin mapping for the job.
|
2005-03-14 23:57:21 +03:00
|
|
|
*/
|
2008-02-28 04:57:57 +03:00
|
|
|
static int orte_rmaps_rr_map(orte_job_t *jdata)
|
2005-03-14 23:57:21 +03:00
|
|
|
{
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_job_map_t *map;
|
|
|
|
orte_app_context_t *app, **apps;
|
2006-10-07 19:45:24 +04:00
|
|
|
orte_std_cntr_t i;
|
2008-02-28 04:57:57 +03:00
|
|
|
opal_list_t node_list, procs;
|
|
|
|
opal_list_item_t *item;
|
|
|
|
orte_node_t *node;
|
2006-12-12 19:07:23 +03:00
|
|
|
orte_vpid_t vpid_start;
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_std_cntr_t num_nodes, num_slots;
|
2006-07-10 18:10:21 +04:00
|
|
|
int rc;
|
2007-01-18 20:15:19 +03:00
|
|
|
orte_std_cntr_t slots_per_node;
|
2005-03-14 23:57:21 +03:00
|
|
|
|
2006-10-07 19:45:24 +04:00
|
|
|
OPAL_TRACE(1);
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* conveniece def */
|
|
|
|
map = jdata->map;
|
|
|
|
apps = (orte_app_context_t**)jdata->apps->addr;
|
2006-10-18 18:01:44 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* start at the beginning... */
|
|
|
|
vpid_start = 0;
|
2005-10-08 02:24:52 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* cycle through the app_contexts, mapping them sequentially */
|
|
|
|
for(i=0; i < jdata->num_apps; i++) {
|
|
|
|
app = apps[i];
|
2006-07-10 18:10:21 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* if the number of processes wasn't specified, then we know there can be only
|
|
|
|
* one app_context allowed in the launch, and that we are to launch it across
|
|
|
|
* all available slots. We'll double-check the single app_context rule first
|
|
|
|
*/
|
|
|
|
if (0 == app->num_procs && 1 < jdata->num_apps) {
|
2006-07-11 06:59:23 +04:00
|
|
|
opal_show_help("help-orte-rmaps-rr.txt", "orte-rmaps-rr:multi-apps-and-zero-np",
|
2008-02-28 04:57:57 +03:00
|
|
|
true, jdata->num_apps, NULL);
|
|
|
|
rc = ORTE_ERR_SILENT;
|
|
|
|
goto error;
|
2006-07-11 06:59:23 +04:00
|
|
|
}
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* for each app_context, we have to get the list of nodes that it can
|
|
|
|
* use since that can now be modified with a hostfile and/or -host
|
|
|
|
* option
|
|
|
|
*/
|
|
|
|
OBJ_CONSTRUCT(&node_list, opal_list_t);
|
|
|
|
if(ORTE_SUCCESS != (rc = orte_rmaps_base_get_target_nodes(&node_list, &num_slots, app,
|
2008-04-10 02:10:53 +04:00
|
|
|
map->policy))) {
|
2008-02-28 04:57:57 +03:00
|
|
|
ORTE_ERROR_LOG(rc);
|
|
|
|
goto error;
|
2006-07-10 18:10:21 +04:00
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
num_nodes = (orte_std_cntr_t)opal_list_get_size(&node_list);
|
|
|
|
|
|
|
|
/* if a bookmark exists from some prior mapping, set us to start there */
|
|
|
|
if (NULL != jdata->bookmark) {
|
|
|
|
cur_node_item = NULL;
|
|
|
|
/* find this node on the list */
|
|
|
|
for (item = opal_list_get_first(&node_list);
|
|
|
|
item != opal_list_get_end(&node_list);
|
|
|
|
item = opal_list_get_next(item)) {
|
|
|
|
node = (orte_node_t*)item;
|
|
|
|
|
|
|
|
if (node->index == jdata->bookmark->index) {
|
|
|
|
cur_node_item = item;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* see if we found it - if not, just start at the beginning */
|
|
|
|
if (NULL == cur_node_item) {
|
|
|
|
cur_node_item = opal_list_get_first(&node_list);
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/* if no bookmark, then just start at the beginning of the list */
|
|
|
|
cur_node_item = opal_list_get_first(&node_list);
|
2006-12-11 21:07:07 +03:00
|
|
|
}
|
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
if (map->pernode && map->npernode == 1) {
|
2006-12-11 21:07:07 +03:00
|
|
|
/* there are three use-cases that we need to deal with:
|
|
|
|
* (a) if -np was not provided, then we just use the number of nodes
|
|
|
|
* (b) if -np was provided AND #procs > #nodes, then error out
|
|
|
|
* (c) if -np was provided AND #procs <= #nodes, then launch
|
|
|
|
* the specified #procs one/node. In this case, we just
|
|
|
|
* leave app->num_procs alone
|
|
|
|
*/
|
2006-07-11 06:59:23 +04:00
|
|
|
if (0 == app->num_procs) {
|
2006-12-11 21:07:07 +03:00
|
|
|
app->num_procs = num_nodes;
|
|
|
|
} else if (app->num_procs > num_nodes) {
|
|
|
|
opal_show_help("help-orte-rmaps-rr.txt", "orte-rmaps-rr:per-node-and-too-many-procs",
|
|
|
|
true, app->num_procs, num_nodes, NULL);
|
2008-02-28 04:57:57 +03:00
|
|
|
rc = ORTE_ERR_SILENT;
|
|
|
|
goto error;
|
2006-12-11 21:07:07 +03:00
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
} else if (map->pernode && map->npernode > 1) {
|
2007-01-18 20:15:19 +03:00
|
|
|
/* first, let's check to see if there are enough slots/node to
|
|
|
|
* meet the request - error out if not
|
|
|
|
*/
|
|
|
|
slots_per_node = num_slots / num_nodes;
|
2008-02-28 04:57:57 +03:00
|
|
|
if (map->npernode > slots_per_node) {
|
2007-01-18 20:15:19 +03:00
|
|
|
opal_show_help("help-orte-rmaps-rr.txt", "orte-rmaps-rr:n-per-node-and-not-enough-slots",
|
2008-02-28 04:57:57 +03:00
|
|
|
true, map->npernode, slots_per_node, NULL);
|
|
|
|
rc = ORTE_ERR_SILENT;
|
|
|
|
goto error;
|
2007-01-18 20:15:19 +03:00
|
|
|
}
|
|
|
|
/* there are three use-cases that we need to deal with:
|
|
|
|
* (a) if -np was not provided, then we just use the n/node * #nodes
|
|
|
|
* (b) if -np was provided AND #procs > (n/node * #nodes), then error out
|
|
|
|
* (c) if -np was provided AND #procs <= (n/node * #nodes), then launch
|
|
|
|
* the specified #procs n/node. In this case, we just
|
|
|
|
* leave app->num_procs alone
|
|
|
|
*/
|
|
|
|
if (0 == app->num_procs) {
|
|
|
|
/* set the num_procs to equal the specified num/node * the number of nodes */
|
2008-02-28 04:57:57 +03:00
|
|
|
app->num_procs = map->npernode * num_nodes;
|
|
|
|
} else if (app->num_procs > (map->npernode * num_nodes)) {
|
2007-01-18 20:15:19 +03:00
|
|
|
opal_show_help("help-orte-rmaps-rr.txt", "orte-rmaps-rr:n-per-node-and-too-many-procs",
|
2008-02-28 04:57:57 +03:00
|
|
|
true, app->num_procs, map->npernode, num_nodes, num_slots, NULL);
|
|
|
|
rc = ORTE_ERR_SILENT;
|
|
|
|
goto error;
|
2007-01-18 20:15:19 +03:00
|
|
|
}
|
2006-12-11 21:07:07 +03:00
|
|
|
} else if (0 == app->num_procs) {
|
|
|
|
/** set the num_procs to equal the number of slots on these mapped nodes - if
|
|
|
|
user has specified "-bynode", then set it to the number of nodes
|
|
|
|
*/
|
2008-04-10 02:10:53 +04:00
|
|
|
if (map->policy & ORTE_RMAPS_BYNODE) {
|
2006-12-11 21:07:07 +03:00
|
|
|
app->num_procs = num_nodes;
|
2008-04-10 02:10:53 +04:00
|
|
|
} else if (map->policy & ORTE_RMAPS_BYSLOT) {
|
2006-12-11 21:07:07 +03:00
|
|
|
app->num_procs = num_slots;
|
2008-04-10 02:10:53 +04:00
|
|
|
} else if (map->policy & ORTE_RMAPS_BYUSER) {
|
2008-02-28 04:57:57 +03:00
|
|
|
/* we can't handle this - it should have been set when we got
|
|
|
|
* the map info. If it wasn't, then we can only error out
|
|
|
|
*/
|
|
|
|
opal_show_help("help-orte-rmaps-rr.txt", "orte-rmaps-rr:no-np-and-user-map",
|
|
|
|
true, app->num_procs, map->npernode, num_nodes, num_slots, NULL);
|
|
|
|
rc = ORTE_ERR_SILENT;
|
|
|
|
goto error;
|
2006-07-11 06:59:23 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/** track the total number of processes we mapped */
|
2008-02-28 04:57:57 +03:00
|
|
|
jdata->num_procs += app->num_procs;
|
2006-07-11 06:59:23 +04:00
|
|
|
|
2005-10-08 02:24:52 +04:00
|
|
|
/* Make assignments */
|
2008-02-28 04:57:57 +03:00
|
|
|
if (map->policy == ORTE_RMAPS_BYUSER) {
|
|
|
|
rc = map_app_by_user_map(app, jdata, vpid_start, &node_list, &procs);
|
|
|
|
} else if (map->policy == ORTE_RMAPS_BYNODE) {
|
|
|
|
rc = map_app_by_node(app, jdata, vpid_start, &node_list);
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
} else {
|
2008-02-28 04:57:57 +03:00
|
|
|
rc = map_app_by_slot(app, jdata, vpid_start, &node_list);
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
}
|
2006-07-10 18:10:21 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* update the starting vpid for the next app_context */
|
|
|
|
vpid_start += app->num_procs;
|
2006-07-10 18:10:21 +04:00
|
|
|
|
(copied from a mail that has a lengthy description of this commit)
I spoke with Tim about this the other day -- he gave me the green
light to go ahead with this, but it turned into a bigger job than I
thought it would be. I revamped how the default RAS scheduling and
round_robin RMAPS mapping occurs. The previous algorithms were pretty
brain dead, and ignored the "slots" and "max_slots" tokens in
hostfiles. I considered this a big enough problem to fix it for the
beta (because there is currently no way to control where processes are
launched on SMPs).
There's still some more bells and whistles that I'd like to implement,
but there's no hurry, and they can go on the trunk at any time. My
patches below are for what I considered "essential", and do the
following:
- honor the "slots" and "max-slots" tokens in the hostfile (and all
their synonyms), meaning that we allocate/map until we fill slots,
and if there are still more processes to allocate/map, we keep going
until we fill max-slots (i.e., only oversubscribe a node if we have
to).
- offer two different algorithms, currently supported by two new
options to orterun. Remember that there are two parts here -- slot
allocation and process mapping. Slot allocation controls how many
processes we'll be running on a node. After that decision has been
made, process mapping effectively controls where the ranks of
MPI_COMM_WORLD (MCW) are placed. Some of the examples given below
don't make sense unless you remember that there is a difference
between the two (which makes total sense, but you have to think
about it in terms of both things):
1. "-bynode": allocates/maps one process per node in a round-robin
fashion until all slots on the node are taken. If we still have more
processes after all slots are taken, then keep going until all
max-slots are taken. Examples:
- The hostfile:
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -bynode -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 2
vogon: MCW ranks 1, 3, 4, 5
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4
vogon: MCW ranks 1, 3, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until each
node's max_slots is hit, of course)
- orterun -bynode -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 2, 4, 6
vogon: MCW ranks 1, 3, 5, 7, 8, 9, 10, 11
2. "-byslot" (this is the default if you don't specify -bynode):
greedily takes all available slots on a node for a job before moving
on to the next node. If we still have processes to allocate/schedule,
then oversubscribe all nodes equally (i.e., go round robin on all
nodes until each node's max_slots is hit). Examples:
- The hostfile
eddie slots=2 max-slots=4
vogon slots=4 max-slots=8
- orterun -np 6 -hostfile hostfile a.out
eddie: MCW ranks 0, 1
vogon: MCW ranks 2, 3, 4, 5
- orterun -np 8 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2
vogon: MCW ranks 3, 4, 5, 6, 7
-> the algorithm oversubscribes all nodes "equally" (until max_slots
is hit)
- orterun -np 12 -hostfile hostfile a.out
eddie: MCW ranks 0, 1, 2, 3
vogon: MCW ranks 4, 5, 6, 7, 8, 9, 10, 11
The above examples are fairly contrived, and it's not clear from them
that you can get different allocation answers in all cases (the
mapping differences are obvious). Consider the following allocation
example:
- The hostfile
eddie count=4
vogon count=4
earth count=4
deep-thought count=4
- orterun -np 8 -hostfile hostfile a.out
eddie: 4 slots will be allocated
vogon: 4 slots will be allocated
earth: no slots allocated
deep-thought: no slots allocated
- orterun -bynode -np 8 -hostfile hostfile a.out
eddie: 2 slots will be allocated
vogon: 2 slots will be allocated
earth: 2 slots will be allocated
deep-thought: 2 slots will be allocated
This commit was SVN r5894.
2005-05-31 20:36:53 +04:00
|
|
|
if (ORTE_SUCCESS != rc) {
|
2006-07-10 18:10:21 +04:00
|
|
|
ORTE_ERROR_LOG(rc);
|
2008-02-28 04:57:57 +03:00
|
|
|
goto error;
|
2005-03-14 23:57:21 +03:00
|
|
|
}
|
2006-07-11 06:59:23 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
/* save the bookmark */
|
|
|
|
jdata->bookmark = (orte_node_t*)cur_node_item;
|
|
|
|
|
|
|
|
/* cleanup the node list - it can differ from one app_context
|
|
|
|
* to another, so we have to get it every time
|
2006-10-19 22:57:29 +04:00
|
|
|
*/
|
2008-02-28 04:57:57 +03:00
|
|
|
while(NULL != (item = opal_list_remove_first(&node_list))) {
|
|
|
|
OBJ_RELEASE(item);
|
2006-10-20 23:00:17 +04:00
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
OBJ_DESTRUCT(&node_list);
|
2005-03-14 23:57:21 +03:00
|
|
|
}
|
|
|
|
|
2006-12-12 19:07:23 +03:00
|
|
|
/* compute and save convenience values */
|
2008-02-28 04:57:57 +03:00
|
|
|
if (ORTE_SUCCESS != (rc = orte_rmaps_base_compute_usage(jdata))) {
|
Compute and pass the local_rank and local number of procs (in that proc's job) on the node.
To be precise, given this hypothetical launching pattern:
host1: vpids 0, 2, 4, 6
host2: vpids 1, 3, 5, 7
The local_rank for these procs would be:
host1: vpids 0->local_rank 0, v2->lr1, v4->lr2, v6->lr3
host2: vpids 1->local_rank 0, v3->lr1, v5->lr2, v7->lr3
and the number of local procs on each node would be four. If vpid=0 then does a comm_spawn of one process on host1, the values of the parent job would remain unchanged. The local_rank of the child process would be 0 and its num_local_procs would be 1 since it is in a separate jobid.
I have verified this functionality for the rsh case - need to verify that slurm and other cases also get the right values. Some consolidation of common code is probably going to occur in the SDS components to make this simpler and more maintainable in the future.
This commit was SVN r14706.
2007-05-21 18:30:10 +04:00
|
|
|
ORTE_ERROR_LOG(rc);
|
2008-02-28 04:57:57 +03:00
|
|
|
return rc;
|
2007-07-14 19:14:07 +04:00
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
|
Bring in the generalized xcast communication system along with the correspondingly revised orted launch. I will send a message out to developers explaining the basic changes. In brief:
1. generalize orte_rml.xcast to become a general broadcast-like messaging system. Messages can now be sent to any tag on the daemons or processes. Note that any message sent via xcast will be delivered to ALL processes in the specified job - you don't get to pick and choose. At a later date, we will introduce an augmented capability that will use the daemons as relays, but will allow you to send to a specified array of process names.
2. extended orte_rml.xcast so it supports more scalable message routing methodologies. At the moment, we support three: (a) direct, which sends the message directly to all recipients; (b) linear, which sends the message to the local daemon on each node, which then relays it to its own local procs; and (b) binomial, which sends the message via a binomial algo across all the daemons, each of which then relays to its own local procs. The crossover points between the algos are adjustable via MCA param, or you can simply demand that a specific algo be used.
3. orteds no longer exhibit two types of behavior: bootproxy or VM. Orteds now always behave like they are part of a virtual machine - they simply launch a job if mpirun tells them to do so. This is another step towards creating an "orteboot" functionality, but also provided a clean system for supporting message relaying.
Note one major impact of this commit: multiple daemons on a node cannot be supported any longer! Only a single daemon/node is now allowed.
This commit is known to break support for the following environments: POE, Xgrid, Xcpu, Windows. It has been tested on rsh, SLURM, and Bproc. Modifications for TM support have been made but could not be verified due to machine problems at LANL. Modifications for SGE have been made but could not be verified. The developers for the non-verified environments will be separately notified along with suggestions on how to fix the problems.
This commit was SVN r15007.
2007-06-12 17:28:54 +04:00
|
|
|
/* define the daemons that we will use for this job */
|
|
|
|
if (ORTE_SUCCESS != (rc = orte_rmaps_base_define_daemons(map))) {
|
|
|
|
ORTE_ERROR_LOG(rc);
|
2008-02-28 04:57:57 +03:00
|
|
|
return rc;
|
2005-03-14 23:57:21 +03:00
|
|
|
}
|
2006-07-10 18:10:21 +04:00
|
|
|
|
2008-02-28 04:57:57 +03:00
|
|
|
return ORTE_SUCCESS;
|
|
|
|
|
|
|
|
error:
|
|
|
|
while(NULL != (item = opal_list_remove_first(&node_list))) {
|
2005-03-14 23:57:21 +03:00
|
|
|
OBJ_RELEASE(item);
|
|
|
|
}
|
2008-02-28 04:57:57 +03:00
|
|
|
OBJ_DESTRUCT(&node_list);
|
2007-03-29 04:50:56 +04:00
|
|
|
|
2005-03-14 23:57:21 +03:00
|
|
|
return rc;
|
|
|
|
}
|
|
|
|
|
|
|
|
orte_rmaps_base_module_t orte_rmaps_round_robin_module = {
|
2008-02-28 04:57:57 +03:00
|
|
|
orte_rmaps_rr_map
|
2005-03-14 23:57:21 +03:00
|
|
|
};
|
|
|
|
|