Fixes trac:967.
A bunch of fixes from the /tmp/iof-fixes branch that fix up ''some'' (but not ''all'') of the problems that we have seen with iof: * Reading very large files via stdin redirected to orteun (Sun saw this) * Reading a little bit of a large file redirected to orterun's stdin and then either closing stdin or exiting the process The Big Change was to make the proxy iof (the one running in non-HNP orteds) send back a "I'm closing the stream" ACK back to the service iof. This tells the HNP that there will be nothing more coming from that peer, and therefore the iof forward should be removed. Many other minor cleanups/fixes, terminology changes, and documentation additions are included in this commit as well. However, there are still some pretty big outstanding issues with IOF that are not addressed either by #967 or this commit. A few examples: * IOF was designed to allow multiple subscribers to a single stream. We're not entirely sure that this works (for one thing, there is nothing in the ORTE/OMPI code base that uses this functionality). * There are also resources leaked when processes/jobs exit (per Ralph's first comment on this ticket). * There is no feedback to close orterun's stdin when all subscribers to the corresponding stream have closed stdin. This commit was SVN r14967. The following Trac tickets were found above: Ticket 967 --> https://svn.open-mpi.org/trac/ompi/ticket/967
Этот коммит содержится в:
Jeff Squyres
родитель
e2dd0a50fc
Коммит
4f3a11b4db
178
orte/mca/iof/README.txt
Обычный файл
178
orte/mca/iof/README.txt
Обычный файл
@ -0,0 +1,178 @@
|
||||
Some notes from Jeff/Rolf while trying to figure out how IOF works...
|
||||
|
||||
1. E-mail from Rolf->Jeff with some pretty ASCII art
|
||||
2. Notes from Jeff about problems we'll have when/if we ever try to
|
||||
use IOF more creatively.
|
||||
|
||||
===============================================================================
|
||||
|
||||
From: Rolf.Vandevaart@Sun.COM
|
||||
Subject: Picture of IOF side of things.
|
||||
Date: June 7, 2007 10:43:42 AM EDT
|
||||
To: jsquyres@cisco.com
|
||||
|
||||
|
||||
Not sure if this will come out for you, but here is an ASCII represantation of what
|
||||
the HNP looks like after IOF has been wired up.
|
||||
|
||||
|
||||
|
||||
VIEW OF DATA STRUCTURES ON IOF SVC SIDE
|
||||
|
||||
KEY: ORTE_IOF_SOURCE=0
|
||||
ORTE_IOF_SINK=1
|
||||
|
||||
After the job starts up, this is how things look on the HNP side of things.
|
||||
|
||||
ENDPOINTS (orte_iof_base_endpoint_t)
|
||||
mode origin tag fd seq ack src_frags sink_frags notes
|
||||
=============================================================================================
|
||||
1 0,0,0 1 1 0 0 0 0 pull() call from rmgr<-
|
||||
============================================================================================= |
|
||||
1 0,0,0 2 2 0 0 0 0 pull() call from rmgr<-|---
|
||||
============================================================================================= | |
|
||||
0 0,0,0 0 0 0 0 0 0 push() call from rmgr | |
|
||||
============================================================================================= | |
|
||||
| |
|
||||
| |
|
||||
SUBSCRIBERS (orte_iof_svc_sub_t) | |
|
||||
ORIGIN TARGET (list) | |
|
||||
name mask tag name mask tag endpoint forward has_been_acked last_ack_forwarded | |
|
||||
====================================================================================== | |
|
||||
0,1,0 2 1 0,0,0 2 1 0 | |
|
||||
ptr-------------------------------------------------| |
|
||||
====================================================================================== |
|
||||
0,1,0 2 2 0,0,0 2 2 0 |
|
||||
ptr-----------------------------------------------------|
|
||||
======================================================================================
|
||||
0,0,0 ff 0 0,1,0 2 0 NULL 1 -----------------------------------
|
||||
|
|
||||
====================================================================================== |
|
||||
|
|
||||
|
|
||||
PUBLISHED (orte_iof_svc_pub_t) |
|
||||
name proxy mask tag endpoint |
|
||||
========================================================================================= |
|
||||
0,1,0 0,0,1 ff 0 NULL <------------------
|
||||
=========================================================================================
|
||||
|
||||
|
||||
FORWARD (orte_iof_svc_fwd_t)
|
||||
This structure is just a connection from a subscriber to publisher. I have
|
||||
omitted it in the drawings. However, it is worth pointing out the structure
|
||||
as I am not clear on why we have the table.
|
||||
|
||||
struct orte_iof_svc_fwd_t {
|
||||
opal_list_item_t super;
|
||||
orte_iof_svc_pub_t* fwd_pub;
|
||||
opal_hash_table_t fwd_seq_hash;
|
||||
};
|
||||
|
||||
|
||||
Note: This first subscriber says that it will receive from any process
|
||||
in the job. Note that the jobid=1 and the mask=2. So, we expect this
|
||||
to collect the stdout from any of the ranks. Obviously the second
|
||||
subscriber says the same thing but for stderr. The third subscriber
|
||||
is for receving data from stdin and sending it out to rank 0 of
|
||||
the job. Notice the mask=ff which means compare cellid,jobid,vpid
|
||||
when addressing where the data goes.
|
||||
|
||||
The first endpoint is created by a call to pull by the rmgr. After
|
||||
the endpoint is created, a subscription is created as well. Then, the
|
||||
subscription is tied to the endpoint.
|
||||
|
||||
For the stdin creation, we first create the subscription, and then the
|
||||
endpoint. In that way, the endpoint is not found and does not get
|
||||
tied to the subscription. Hmmm, this I do not really understand.
|
||||
|
||||
|
||||
|
||||
APPENDIX A
|
||||
These are the defines that go with the mask.
|
||||
#define ORTE_NS_CMP_NONE 0x00
|
||||
#define ORTE_NS_CMP_CELLID 0x01
|
||||
#define ORTE_NS_CMP_JOBID 0x02
|
||||
#define ORTE_NS_CMP_VPID 0x04
|
||||
#define ORTE_NS_CMP_ALL 0Xff
|
||||
|
||||
|
||||
When we get a HDR_MSG, we call orte_iof_svc_proxy_msg()
|
||||
|
||||
APPENDIX B
|
||||
There are two dbx files that help get to where we want to get
|
||||
for seeing how things work.
|
||||
start.x : Run this first to get initial breakpoint. Needs this
|
||||
so we can set additional breakpoints. This also has some very
|
||||
helpful aliases for looking at the structures shown above.
|
||||
|
||||
follow.x : Run this second to set initial breakpoints and setup
|
||||
some useful aliases.
|
||||
|
||||
===============================================================================
|
||||
|
||||
Random notes from Jeff:
|
||||
|
||||
- Many issues may not come up because we only have single subscribers;
|
||||
I'm sure new things will come up. Examples:
|
||||
|
||||
- What happens if all subscribers to a stream disconnect, and then a
|
||||
new subscriber connects? I'm guessing the ACKs will be all
|
||||
screwed up and we'll never end up reading from that fd again
|
||||
(because it will likely be stalled because not enough acks have
|
||||
been received, and therefore it removed itself from the event
|
||||
engine).
|
||||
|
||||
- If all subscribers disconnect from a stdin/SINK, chances are that
|
||||
we'll lose the last frag that was sent before the disconnect.
|
||||
I.e., if there was a frag in flight when the disconnect was
|
||||
received, that frag is effectively lost. So if someone reconnects
|
||||
to the stdin stream later, it won't start reading exactly where
|
||||
the first subscriber left off. We need to define what is
|
||||
*supposed* to happen here...
|
||||
|
||||
- odls default: make handling of vpid 0 uniform between setup and
|
||||
takedown -- some kind of global variable, perhaps? (not multi-proc /
|
||||
thread safe)
|
||||
|
||||
- odls default: currently, we publish stdin (if relevant), stdout, and
|
||||
stderr (note that only the stdin publish message gets sent to svc;
|
||||
the publish for SOURCEs stdout/stderr is not actually sent to the
|
||||
svc because all SOURCE frags are sent to the svc automatically).
|
||||
But we only unpublish stdout. I think we should either:
|
||||
- publish stdin, stdout, stderr, and unpublish stdin, stdout, stderr
|
||||
or
|
||||
- publish stdin, and unpublish stdin
|
||||
I.e., make the code symmetric.
|
||||
|
||||
Note, however, that unpublish for STDOUT/STDERR are sent to the svc
|
||||
(whereas publish for STDOUT/STDERR are not). So if we unpublish
|
||||
stdout/stderr, we'll be creating a storm to the svc upon shutdown
|
||||
(i.e,. scalability problems). :-(
|
||||
|
||||
- for scalability, we want to be able to change the proxy to *not*
|
||||
unconfitionally send everything to svc. But this has the problem
|
||||
that if we do this, then we have to send the publish request to the
|
||||
svc (which we don't today since everything just automatically goes
|
||||
to svc). But then in the common case (where vpid!=0 has no
|
||||
stdout/stderr), we're flooding svc with N publish requests from all
|
||||
the vpids, simply creating a different scalability problem (during
|
||||
startup).
|
||||
|
||||
- random q: are the proxy publish requests not sent back to svc
|
||||
because it prevents a storm of publish requests during startup?
|
||||
I.e., this was intentional to give better scalability? Could be;
|
||||
but it still seems weird...
|
||||
|
||||
Perhaps a better scheme would be to have the IOF *assume* that the
|
||||
stdin/stdout/stderr are all published upon startup (or be told by a
|
||||
single control message; perhaps in the app context?) and further
|
||||
*assume* that they are all unpublished when the job completes.
|
||||
|
||||
Putting this info in the app context (for example) might jive with a
|
||||
more capable orterun that allows flexible stdin/stdout/stderr
|
||||
mapping (think: mpirun --screen ...). mpirun makes the decision
|
||||
about how to wire up stdin/stdout/stderr and includes it in the app
|
||||
context (or whatever). This is given to the svc who then creates
|
||||
publications as relevant. Upon job completion, all
|
||||
publications/subscriptions related to that job are destroyed.
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
||||
* Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The University of Tennessee and The University
|
||||
@ -36,7 +36,7 @@ int orte_iof_base_close(void)
|
||||
selected */
|
||||
|
||||
if (orte_iof_base.iof_flush) {
|
||||
orte_iof_base_flush();
|
||||
orte_iof.iof_flush();
|
||||
orte_iof_base.iof_flush = false;
|
||||
}
|
||||
|
||||
@ -56,12 +56,19 @@ int orte_iof_base_close(void)
|
||||
while((item = opal_list_remove_first(&orte_iof_base.iof_endpoints)) != NULL) {
|
||||
OBJ_RELEASE(item);
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
|
||||
if (NULL != orte_iof_base.iof_service) {
|
||||
free(orte_iof_base.iof_service);
|
||||
}
|
||||
|
||||
OBJ_DESTRUCT(&orte_iof_base.iof_components_opened);
|
||||
OBJ_DESTRUCT(&orte_iof_base.iof_endpoints);
|
||||
OBJ_DESTRUCT(&orte_iof_base.iof_lock);
|
||||
OBJ_DESTRUCT(&orte_iof_base.iof_condition);
|
||||
OBJ_DESTRUCT(&orte_iof_base.iof_fragments);
|
||||
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
|
||||
@ -9,6 +9,8 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* Copyright (c) 2007 Sun Microsystems, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -50,6 +52,21 @@
|
||||
#include "orte/mca/iof/base/iof_base_endpoint.h"
|
||||
#include "orte/mca/iof/base/iof_base_fragment.h"
|
||||
|
||||
|
||||
/*
|
||||
* Globals
|
||||
*/
|
||||
static bool sigpipe_event_initialized = false;
|
||||
static struct opal_event sigpipe_event;
|
||||
|
||||
static void sigpipe_signal_callback(int fd, short event, void *arg)
|
||||
{
|
||||
/* Do nothing -- the purpose of this handler is so that we don't
|
||||
die due to SIGPIPE, but we don't need to *do* anything in this
|
||||
handler. */
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Construct/Destructor
|
||||
*/
|
||||
@ -61,7 +78,8 @@ static void orte_iof_base_endpoint_construct(orte_iof_base_endpoint_t* endpoint)
|
||||
endpoint->ep_ack = 0;
|
||||
endpoint->ep_fd = -1;
|
||||
memset(&endpoint->ep_event,0,sizeof(endpoint->ep_event));
|
||||
OBJ_CONSTRUCT(&endpoint->ep_frags, opal_list_t);
|
||||
OBJ_CONSTRUCT(&endpoint->ep_source_frags, opal_list_t);
|
||||
OBJ_CONSTRUCT(&endpoint->ep_sink_frags, opal_list_t);
|
||||
OBJ_CONSTRUCT(&endpoint->ep_callbacks, opal_list_t);
|
||||
}
|
||||
|
||||
@ -70,7 +88,8 @@ static void orte_iof_base_endpoint_destruct(orte_iof_base_endpoint_t* endpoint)
|
||||
if(endpoint->ep_fd >= 0) {
|
||||
opal_event_del(&endpoint->ep_event);
|
||||
}
|
||||
OBJ_DESTRUCT(&endpoint->ep_frags);
|
||||
OBJ_DESTRUCT(&endpoint->ep_source_frags);
|
||||
OBJ_DESTRUCT(&endpoint->ep_sink_frags);
|
||||
OBJ_DESTRUCT(&endpoint->ep_callbacks);
|
||||
}
|
||||
|
||||
@ -83,29 +102,22 @@ OBJ_CLASS_INSTANCE(
|
||||
/**
|
||||
* Construct/Destructor
|
||||
*/
|
||||
|
||||
static void orte_iof_base_callback_construct(orte_iof_base_callback_t* cb)
|
||||
{
|
||||
cb->cb_func = 0;
|
||||
cb->cb_data = NULL;
|
||||
}
|
||||
|
||||
static void orte_iof_base_callback_destruct(orte_iof_base_callback_t* cb)
|
||||
{
|
||||
}
|
||||
|
||||
OBJ_CLASS_INSTANCE(
|
||||
orte_iof_base_callback_t,
|
||||
opal_list_item_t,
|
||||
orte_iof_base_callback_construct,
|
||||
orte_iof_base_callback_destruct);
|
||||
|
||||
|
||||
NULL);
|
||||
|
||||
/*
|
||||
* Callback when non-blocking OOB send completes.
|
||||
* Callback when non-blocking RML send completes.
|
||||
*/
|
||||
|
||||
static void orte_iof_base_endpoint_send_cb(
|
||||
int status,
|
||||
orte_process_name_t* peer,
|
||||
@ -116,7 +128,9 @@ static void orte_iof_base_endpoint_send_cb(
|
||||
{
|
||||
orte_iof_base_frag_t* frag = (orte_iof_base_frag_t*)cbdata;
|
||||
orte_iof_base_endpoint_t* endpoint = frag->frag_owner;
|
||||
opal_list_remove_item(&endpoint->ep_frags, &frag->super.super);
|
||||
opal_list_remove_item(&endpoint->ep_source_frags, &frag->super.super);
|
||||
opal_output(orte_iof_base.iof_output, "iof_base_endpoint: send cb, source_frags list len: %d",
|
||||
(int) opal_list_get_size(&endpoint->ep_source_frags));
|
||||
ORTE_IOF_BASE_FRAG_RETURN(frag);
|
||||
|
||||
/* Decrement the refcount on the endpoint; matches the RETAIN for
|
||||
@ -140,6 +154,7 @@ static void orte_iof_base_endpoint_read_handler(int fd, short flags, void *cbdat
|
||||
/* allocate a fragment */
|
||||
ORTE_IOF_BASE_FRAG_ALLOC(frag,rc);
|
||||
if(NULL == frag) {
|
||||
/* JMS shouldn't we do something here? */
|
||||
return;
|
||||
}
|
||||
|
||||
@ -156,30 +171,39 @@ static void orte_iof_base_endpoint_read_handler(int fd, short flags, void *cbdat
|
||||
rc = (int)readed;
|
||||
}
|
||||
#endif /* !defined(__WINDOWS__) */
|
||||
if(rc < 0) {
|
||||
if (rc < 0) {
|
||||
/* non-blocking, retry */
|
||||
if(errno == EAGAIN || errno == EINTR) {
|
||||
if (EAGAIN == errno || EINTR == errno) {
|
||||
ORTE_IOF_BASE_FRAG_RETURN(frag);
|
||||
/* error on the connection */
|
||||
} else {
|
||||
orte_iof_base_endpoint_closed(endpoint);
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return;
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return;
|
||||
}
|
||||
|
||||
/* Error on the connection */
|
||||
orte_iof_base_endpoint_closed(endpoint);
|
||||
/* Fall through to send 0 byte message to other side
|
||||
indicating that the endpoint is now closed. */
|
||||
rc = 0;
|
||||
} else if (rc == 0) {
|
||||
/* peer has closed connection - send 0 byte message to subscribers */
|
||||
/* peer has closed connection (will fall through to send a 0
|
||||
byte message, therefore telling the RML side that the fd
|
||||
side has closed its connection) */
|
||||
orte_iof_base_endpoint_closed(endpoint);
|
||||
}
|
||||
|
||||
/* Do not append the fragment before we know that we have some data (even 0 bytes it's OK) */
|
||||
/* Do not append the fragment before we know that we have some
|
||||
data (even a 0 byte mesage is OK -- that indicates that the
|
||||
file descriptor has closed) */
|
||||
frag->frag_owner = endpoint;
|
||||
opal_list_append(&endpoint->ep_frags, &frag->super.super);
|
||||
opal_list_append(&endpoint->ep_source_frags, &frag->super.super);
|
||||
opal_output(orte_iof_base.iof_output, "iof_base_endpoint: read handler, source_frags list len: %d",
|
||||
(int) opal_list_get_size(&endpoint->ep_source_frags));
|
||||
frag->frag_iov[1].iov_len = frag->frag_len = rc;
|
||||
|
||||
/* fill in the header */
|
||||
hdr = &frag->frag_hdr;
|
||||
hdr->hdr_common.hdr_type = ORTE_IOF_BASE_HDR_MSG;
|
||||
hdr->hdr_msg.msg_src = endpoint->ep_name;
|
||||
hdr->hdr_msg.msg_origin = endpoint->ep_origin;
|
||||
hdr->hdr_msg.msg_proxy = *ORTE_PROC_MY_NAME;
|
||||
hdr->hdr_msg.msg_tag = endpoint->ep_tag;
|
||||
hdr->hdr_msg.msg_seq = endpoint->ep_seq;
|
||||
@ -189,6 +213,7 @@ static void orte_iof_base_endpoint_read_handler(int fd, short flags, void *cbdat
|
||||
/* if window size has been exceeded - disable forwarding */
|
||||
endpoint->ep_seq += frag->frag_len;
|
||||
if(ORTE_IOF_BASE_SEQDIFF(endpoint->ep_seq,endpoint->ep_ack) > orte_iof_base.iof_window_size) {
|
||||
opal_output(orte_iof_base.iof_output, "iof_base_endpoint read handler: window exceeded -- reading disabled");
|
||||
opal_event_del(&endpoint->ep_event);
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
@ -197,7 +222,7 @@ static void orte_iof_base_endpoint_read_handler(int fd, short flags, void *cbdat
|
||||
deleted before the frag */
|
||||
OBJ_RETAIN(endpoint);
|
||||
|
||||
/* start non-blocking OOB call to forward received data */
|
||||
/* start non-blocking RML call to forward received data */
|
||||
rc = orte_rml.send_nb(
|
||||
orte_iof_base.iof_service,
|
||||
frag->frag_iov,
|
||||
@ -215,6 +240,7 @@ static void orte_iof_base_endpoint_read_handler(int fd, short flags, void *cbdat
|
||||
|
||||
static void orte_iof_base_endpoint_write_handler(int sd, short flags, void *user)
|
||||
{
|
||||
int errno_save;
|
||||
orte_iof_base_endpoint_t* endpoint = (orte_iof_base_endpoint_t*)user;
|
||||
|
||||
/*
|
||||
@ -222,8 +248,8 @@ static void orte_iof_base_endpoint_write_handler(int sd, short flags, void *user
|
||||
* until the output descriptor would block
|
||||
*/
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
while(opal_list_get_size(&endpoint->ep_frags)) {
|
||||
orte_iof_base_frag_t* frag = (orte_iof_base_frag_t*)opal_list_get_first(&endpoint->ep_frags);
|
||||
while(opal_list_get_size(&endpoint->ep_sink_frags)) {
|
||||
orte_iof_base_frag_t* frag = (orte_iof_base_frag_t*)opal_list_get_first(&endpoint->ep_sink_frags);
|
||||
int rc;
|
||||
|
||||
/* close connection on zero byte message */
|
||||
@ -235,12 +261,23 @@ static void orte_iof_base_endpoint_write_handler(int sd, short flags, void *user
|
||||
|
||||
/* progress pending messages */
|
||||
rc = write(endpoint->ep_fd, frag->frag_ptr, frag->frag_len);
|
||||
if(rc < 0) {
|
||||
if(errno == EAGAIN)
|
||||
errno_save = errno;
|
||||
if (rc < 0) {
|
||||
if (EAGAIN == errno_save) {
|
||||
break;
|
||||
if(errno == EINTR)
|
||||
continue;
|
||||
}
|
||||
if (EINTR == errno_save) {
|
||||
continue;
|
||||
}
|
||||
/* All other errors -- to include sigpipe -- mean that
|
||||
Something Bad happened and we should abort in
|
||||
despair. */
|
||||
orte_iof_base_endpoint_closed(endpoint);
|
||||
|
||||
/* Send a ACK-AND-CLOSE back to the service so that it
|
||||
knows not to wait for any further ACKs */
|
||||
orte_iof_base_frag_ack(frag, true);
|
||||
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return;
|
||||
}
|
||||
@ -249,14 +286,14 @@ static void orte_iof_base_endpoint_write_handler(int sd, short flags, void *user
|
||||
if(frag->frag_len > 0) {
|
||||
break;
|
||||
}
|
||||
opal_list_remove_item(&endpoint->ep_frags, &frag->super.super);
|
||||
opal_list_remove_item(&endpoint->ep_sink_frags, &frag->super.super);
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
orte_iof_base_frag_ack(frag);
|
||||
orte_iof_base_frag_ack(frag, false);
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
}
|
||||
|
||||
/* is there anything left to write? */
|
||||
if(opal_list_get_size(&endpoint->ep_frags) == 0) {
|
||||
if(opal_list_get_size(&endpoint->ep_sink_frags) == 0) {
|
||||
opal_event_del(&endpoint->ep_event);
|
||||
if(orte_iof_base.iof_waiting) {
|
||||
opal_condition_signal(&orte_iof_base.iof_condition);
|
||||
@ -306,7 +343,7 @@ static orte_iof_base_endpoint_t* orte_iof_base_endpoint_lookup(
|
||||
item != opal_list_get_end(&orte_iof_base.iof_endpoints);
|
||||
item = opal_list_get_next(item)) {
|
||||
orte_iof_base_endpoint_t* endpoint = (orte_iof_base_endpoint_t*)item;
|
||||
if(orte_ns.compare_fields(ORTE_NS_CMP_ALL,proc,&endpoint->ep_name) == 0 &&
|
||||
if(orte_ns.compare_fields(ORTE_NS_CMP_ALL,proc,&endpoint->ep_origin) == 0 &&
|
||||
endpoint->ep_tag == tag && endpoint->ep_mode == mode) {
|
||||
OBJ_RETAIN(endpoint);
|
||||
return endpoint;
|
||||
@ -331,8 +368,17 @@ int orte_iof_base_endpoint_create(
|
||||
int rc;
|
||||
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
|
||||
/* If we haven't initialized the event yet, do so now */
|
||||
if (!sigpipe_event_initialized) {
|
||||
opal_signal_set(&sigpipe_event, SIGPIPE,
|
||||
sigpipe_signal_callback, &sigpipe_event);
|
||||
opal_signal_add(&sigpipe_event, NULL);
|
||||
sigpipe_event_initialized = true;
|
||||
}
|
||||
|
||||
if((endpoint = orte_iof_base_endpoint_lookup(proc,mode,tag)) != NULL) {
|
||||
OBJ_RELEASE(endpoint);
|
||||
OBJ_RETAIN(endpoint);
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
@ -341,7 +387,7 @@ int orte_iof_base_endpoint_create(
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return ORTE_ERR_OUT_OF_RESOURCE;
|
||||
}
|
||||
endpoint->ep_name = *proc;
|
||||
endpoint->ep_origin = *proc;
|
||||
endpoint->ep_mode = mode;
|
||||
endpoint->ep_tag = tag;
|
||||
endpoint->ep_fd = fd;
|
||||
@ -365,7 +411,7 @@ int orte_iof_base_endpoint_create(
|
||||
(ORTE_IOF_SINK == mode && ORTE_IOF_STDOUT == tag && 1 == fd) ||
|
||||
(ORTE_IOF_SINK == mode && ORTE_IOF_STDERR == tag && 2 == fd))) {
|
||||
if((flags = fcntl(fd, F_GETFL, 0)) < 0) {
|
||||
opal_output(0, "[%s:%d]: fcntl(F_GETFL) failed with errno=%d\n",
|
||||
opal_output(orte_iof_base.iof_output, "[%s:%d]: fcntl(F_GETFL) failed with errno=%d\n",
|
||||
__FILE__, __LINE__, errno);
|
||||
} else {
|
||||
flags |= O_NONBLOCK;
|
||||
@ -407,6 +453,7 @@ int orte_iof_base_endpoint_create(
|
||||
}
|
||||
break;
|
||||
case ORTE_IOF_SINK:
|
||||
/* Create the event for use later; don't add it now */
|
||||
opal_event_set(
|
||||
&endpoint->ep_event,
|
||||
endpoint->ep_fd,
|
||||
@ -415,7 +462,7 @@ int orte_iof_base_endpoint_create(
|
||||
endpoint);
|
||||
break;
|
||||
default:
|
||||
opal_output(0, "orte_iof_base_endpoint_create: invalid mode %d\n", mode);
|
||||
opal_output(orte_iof_base.iof_output, "orte_iof_base_endpoint_create: invalid mode %d\n", mode);
|
||||
return ORTE_ERR_BAD_PARAM;
|
||||
}
|
||||
|
||||
@ -440,10 +487,12 @@ int orte_iof_base_endpoint_delete(
|
||||
while(item != opal_list_get_end(&orte_iof_base.iof_endpoints)) {
|
||||
opal_list_item_t* next = opal_list_get_next(item);
|
||||
orte_iof_base_endpoint_t* endpoint = (orte_iof_base_endpoint_t*)item;
|
||||
if(orte_ns.compare_fields(mask,proc,&endpoint->ep_name) == 0 &&
|
||||
if(orte_ns.compare_fields(mask,proc,&endpoint->ep_origin) == 0 &&
|
||||
endpoint->ep_tag == tag) {
|
||||
OBJ_RELEASE(endpoint);
|
||||
opal_list_remove_item(&orte_iof_base.iof_endpoints,&endpoint->super);
|
||||
OBJ_RELEASE(endpoint);
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
item = next;
|
||||
}
|
||||
@ -457,6 +506,19 @@ int orte_iof_base_endpoint_delete(
|
||||
|
||||
void orte_iof_base_endpoint_closed(orte_iof_base_endpoint_t* endpoint)
|
||||
{
|
||||
/* For sinks: discard any fragments that were waiting to be
|
||||
written down the fd (because the process on the other side of
|
||||
the fd is no longer there -- we're just about to close the
|
||||
fd). */
|
||||
if (ORTE_IOF_SINK == endpoint->ep_mode) {
|
||||
while (NULL != opal_list_remove_first(&(endpoint->ep_sink_frags))){
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Upper layer will take care of signaling any waiting
|
||||
condition variable -- no need to do it here */
|
||||
}
|
||||
|
||||
/* Special case: if we're a sink and one of the special streams
|
||||
(stdout or stderr), don't close anything because we don't want
|
||||
to *actually* close stdout or stderr just because a remote
|
||||
@ -470,16 +532,7 @@ void orte_iof_base_endpoint_closed(orte_iof_base_endpoint_t* endpoint)
|
||||
}
|
||||
|
||||
/* remove any event handlers */
|
||||
switch(endpoint->ep_mode) {
|
||||
case ORTE_IOF_SOURCE:
|
||||
opal_event_del(&endpoint->ep_event);
|
||||
break;
|
||||
case ORTE_IOF_SINK:
|
||||
if(opal_list_get_size(&endpoint->ep_frags)) {
|
||||
opal_event_del(&endpoint->ep_event);
|
||||
}
|
||||
break;
|
||||
}
|
||||
opal_event_del(&endpoint->ep_event);
|
||||
|
||||
/* close associated file descriptor */
|
||||
close(endpoint->ep_fd);
|
||||
@ -491,9 +544,9 @@ void orte_iof_base_endpoint_closed(orte_iof_base_endpoint_t* endpoint)
|
||||
*/
|
||||
|
||||
orte_iof_base_endpoint_t* orte_iof_base_endpoint_match(
|
||||
const orte_process_name_t* dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
int dst_tag)
|
||||
const orte_process_name_t* target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
int target_tag)
|
||||
{
|
||||
opal_list_item_t* item;
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
@ -501,8 +554,10 @@ orte_iof_base_endpoint_t* orte_iof_base_endpoint_match(
|
||||
item != opal_list_get_end(&orte_iof_base.iof_endpoints);
|
||||
item = opal_list_get_next(item)) {
|
||||
orte_iof_base_endpoint_t* endpoint = (orte_iof_base_endpoint_t*)item;
|
||||
if(orte_ns.compare_fields(dst_mask,dst_name,&endpoint->ep_name) == 0) {
|
||||
if(endpoint->ep_tag == dst_tag || endpoint->ep_tag == ORTE_IOF_ANY || dst_tag == ORTE_IOF_ANY) {
|
||||
if(orte_ns.compare_fields(target_mask,target_name,&endpoint->ep_origin) == 0) {
|
||||
if(endpoint->ep_tag == target_tag ||
|
||||
endpoint->ep_tag == ORTE_IOF_ANY ||
|
||||
target_tag == ORTE_IOF_ANY) {
|
||||
OBJ_RETAIN(endpoint);
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return endpoint;
|
||||
@ -521,7 +576,7 @@ orte_iof_base_endpoint_t* orte_iof_base_endpoint_match(
|
||||
|
||||
int orte_iof_base_endpoint_forward(
|
||||
orte_iof_base_endpoint_t* endpoint,
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_msg_header_t* hdr,
|
||||
const unsigned char* data)
|
||||
{
|
||||
@ -542,7 +597,7 @@ int orte_iof_base_endpoint_forward(
|
||||
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
frag->frag_owner = endpoint;
|
||||
frag->frag_src = *src;
|
||||
frag->frag_src = *origin;
|
||||
frag->frag_hdr.hdr_msg = *hdr;
|
||||
frag->frag_len = len;
|
||||
|
||||
@ -552,7 +607,7 @@ int orte_iof_base_endpoint_forward(
|
||||
item = opal_list_get_next(item)) {
|
||||
orte_iof_base_callback_t* cb = (orte_iof_base_callback_t*)item;
|
||||
cb->cb_func(
|
||||
&hdr->msg_src,
|
||||
&hdr->msg_origin,
|
||||
hdr->msg_tag,
|
||||
cb->cb_data,
|
||||
data,
|
||||
@ -563,8 +618,11 @@ int orte_iof_base_endpoint_forward(
|
||||
|
||||
/* try to write w/out copying data */
|
||||
|
||||
if(opal_list_get_size(&endpoint->ep_frags) == 0) {
|
||||
if(opal_list_get_size(&endpoint->ep_sink_frags) == 0) {
|
||||
if(len == 0) {
|
||||
/* No ACK required because the frag is of 0 length
|
||||
(ACKs are based on fragment length; an ACK of 0
|
||||
bytes would do nothing) */
|
||||
ORTE_IOF_BASE_FRAG_RETURN(frag);
|
||||
orte_iof_base_endpoint_closed(endpoint);
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
@ -573,8 +631,13 @@ int orte_iof_base_endpoint_forward(
|
||||
rc = write(endpoint->ep_fd,data,len);
|
||||
if(rc < 0) {
|
||||
if (errno != EAGAIN && errno != EINTR) {
|
||||
ORTE_IOF_BASE_FRAG_RETURN(frag);
|
||||
orte_iof_base_endpoint_closed(endpoint);
|
||||
|
||||
/* Send a ACK-AND-CLOSE back to the service so
|
||||
that it knows not to wait for any further
|
||||
ACKs */
|
||||
orte_iof_base_frag_ack(frag, true);
|
||||
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
@ -583,27 +646,34 @@ int orte_iof_base_endpoint_forward(
|
||||
frag->frag_len -= rc;
|
||||
}
|
||||
|
||||
if(frag->frag_len > 0) {
|
||||
/* Ensure to handle both cases:
|
||||
1. When ep_sink_frags was not empty (regardless of frag_len)
|
||||
2. When ep_sink_frags was empty, but we fell through from above */
|
||||
if(frag->frag_len > 0 || 0 == len) {
|
||||
/* handle incomplete write - also queue up 0 byte message
|
||||
* and recognize this as a request to close the descriptor
|
||||
* when all pending operations complete
|
||||
*/
|
||||
frag->frag_ptr = frag->frag_data;
|
||||
memcpy(frag->frag_ptr, data+rc, frag->frag_len);
|
||||
opal_list_append(&endpoint->ep_frags, &frag->super.super);
|
||||
if(opal_list_get_size(&endpoint->ep_frags) == 1) {
|
||||
opal_list_append(&endpoint->ep_sink_frags, &frag->super.super);
|
||||
/* If we're the first frag to be put on the sink_frags
|
||||
list, then enable the event that will tell us when the
|
||||
fd becomes writeable */
|
||||
if(opal_list_get_size(&endpoint->ep_sink_frags) == 1) {
|
||||
opal_output(orte_iof_base.iof_output, "iof_base_endpoint forwarding frag; re-enabled reading for endpoint");
|
||||
opal_event_add(&endpoint->ep_event,0);
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
} else {
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
/* acknowledge fragment */
|
||||
orte_iof_base_frag_ack(frag);
|
||||
orte_iof_base_frag_ack(frag, false);
|
||||
}
|
||||
} else {
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
/* acknowledge fragment */
|
||||
orte_iof_base_frag_ack(frag);
|
||||
orte_iof_base_frag_ack(frag, false);
|
||||
}
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
@ -631,7 +701,7 @@ int orte_iof_base_callback_create(
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return ORTE_ERR_OUT_OF_RESOURCE;
|
||||
}
|
||||
endpoint->ep_name = *proc;
|
||||
endpoint->ep_origin = *proc;
|
||||
endpoint->ep_mode = ORTE_IOF_SINK;
|
||||
endpoint->ep_tag = tag;
|
||||
endpoint->ep_fd = -1;
|
||||
@ -693,7 +763,7 @@ int orte_iof_base_endpoint_ack(
|
||||
endpoint->ep_ack = seq;
|
||||
window_open =
|
||||
ORTE_IOF_BASE_SEQDIFF(endpoint->ep_seq,endpoint->ep_ack) < orte_iof_base.iof_window_size;
|
||||
|
||||
|
||||
/* someone is waiting on all output to be flushed */
|
||||
if(orte_iof_base.iof_waiting && endpoint->ep_seq == endpoint->ep_ack) {
|
||||
opal_condition_signal(&orte_iof_base.iof_condition);
|
||||
@ -701,9 +771,37 @@ int orte_iof_base_endpoint_ack(
|
||||
|
||||
/* check to see if we need to reenable forwarding */
|
||||
if(window_closed && window_open) {
|
||||
opal_output(orte_iof_base.iof_output, "iof_base_endpoint ack; re-enabled reading for endpoint");
|
||||
opal_event_add(&endpoint->ep_event, 0);
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* See description in iof_base_endpoint.h
|
||||
*/
|
||||
bool orte_iof_base_endpoint_have_pending_frags(
|
||||
orte_iof_base_endpoint_t* endpoint)
|
||||
{
|
||||
if (ORTE_IOF_SOURCE == endpoint->ep_mode) {
|
||||
return !opal_list_is_empty(&endpoint->ep_source_frags);
|
||||
} else {
|
||||
return !opal_list_is_empty(&endpoint->ep_sink_frags);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* See description in iof_base_endpoint.h
|
||||
*/
|
||||
bool orte_iof_base_endpoint_have_pending_acks(
|
||||
orte_iof_base_endpoint_t* endpoint)
|
||||
{
|
||||
if (ORTE_IOF_SOURCE == endpoint->ep_mode) {
|
||||
return (endpoint->ep_seq == endpoint->ep_ack);
|
||||
} else {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
@ -9,6 +9,8 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Sun Microsystems, Inc. All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -25,9 +27,7 @@
|
||||
#include "orte/mca/iof/iof.h"
|
||||
#include "orte/mca/iof/base/iof_base_header.h"
|
||||
|
||||
#if defined(c_plusplus) || defined(__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
BEGIN_C_DECLS
|
||||
|
||||
/**
|
||||
* Structure store callbacks
|
||||
@ -47,16 +47,39 @@ ORTE_DECLSPEC OBJ_CLASS_DECLARATION(orte_iof_base_callback_t);
|
||||
*/
|
||||
|
||||
struct orte_iof_base_endpoint_t {
|
||||
/** Parent */
|
||||
opal_list_item_t super;
|
||||
/** ORTE_IOF_SOURCE or ORTE_IOF_SINK */
|
||||
orte_iof_base_mode_t ep_mode;
|
||||
orte_process_name_t ep_name;
|
||||
/** The origin process for this endpoint. Will either by myself
|
||||
(i.e., it's an fd that represents a source or a sink in my
|
||||
process) or another process (i.e., this process is acting as a
|
||||
proxy for another process and [typically] has a pipe/fd optn
|
||||
to that process to get their stdin, stdout, or stderr). */
|
||||
orte_process_name_t ep_origin;
|
||||
/** Predefined tags: ORTE_IOF_ANY, ORTE_IOF_STDIN, ORTE_IOF_STDOUT,
|
||||
ORTE_IOF_STDERR */
|
||||
int ep_tag;
|
||||
/** File descriptor to read or write from (or -1 if it has been
|
||||
closed */
|
||||
int ep_fd;
|
||||
/** Rollover byte count of what has been forwarded from the fd to
|
||||
other targets */
|
||||
uint32_t ep_seq;
|
||||
/** Minimum byte count of what has been ACK'ed from all the targets
|
||||
that are listening to this endpoint */
|
||||
uint32_t ep_ack;
|
||||
/** Event library event for this file descriptor */
|
||||
opal_event_t ep_event;
|
||||
/** Special event library event for the case of stdin */
|
||||
opal_event_t ep_stdin_event;
|
||||
opal_list_t ep_frags;
|
||||
/** The list for fragments that are in-flight from a SOURCE
|
||||
endpoint */
|
||||
opal_list_t ep_source_frags;
|
||||
/** The list for fragments that are in-flight from a SINK
|
||||
endpoint */
|
||||
opal_list_t ep_sink_frags;
|
||||
/** List of callbacks for subscriptions */
|
||||
opal_list_t ep_callbacks;
|
||||
};
|
||||
typedef struct orte_iof_base_endpoint_t orte_iof_base_endpoint_t;
|
||||
@ -74,10 +97,14 @@ ORTE_DECLSPEC OBJ_CLASS_DECLARATION(orte_iof_base_endpoint_t);
|
||||
/**
|
||||
* Create a local endpoint.
|
||||
*
|
||||
* @param name Process name corresponding to endpoint.
|
||||
* @param name Origin process name corresponding to endpoint.
|
||||
* @param mode Source or sink of data (exclusive).
|
||||
* @param tag Logical tag for matching.
|
||||
* @aram fd Local file descriptor corresponding to endpoint.
|
||||
* @param fd Local file descriptor corresponding to endpoint. If the
|
||||
* endpoint originates in this process, it'll be an fd in this
|
||||
* process. If this process is acting as a proxy for another process,
|
||||
* then the fd will be a pipe to that other process (e.g., the origin
|
||||
* process' stdin, stdout, or stderr).
|
||||
*/
|
||||
|
||||
ORTE_DECLSPEC int orte_iof_base_endpoint_create(
|
||||
@ -106,10 +133,10 @@ ORTE_DECLSPEC int orte_iof_base_callback_delete(
|
||||
|
||||
|
||||
/**
|
||||
* Delete all local endpoints matching the specified
|
||||
* name/mask/tag parameters.
|
||||
* Delete all local endpoints matching the specified origin / mask /
|
||||
* tag parameters.
|
||||
*
|
||||
* @paran name Process name corresponding to one or more endpoint(s).
|
||||
* @paran name Origin process name corresponding to one or more endpoint(s).
|
||||
* @param mask Mask used for name comparisons.
|
||||
* @param tag Tag for matching endpoints.
|
||||
*/
|
||||
@ -127,14 +154,14 @@ ORTE_DECLSPEC int orte_iof_base_endpoint_close(
|
||||
orte_iof_base_endpoint_t* endpoint);
|
||||
|
||||
/**
|
||||
* Attempt to match an endpoint based on the destination
|
||||
* process name/mask/tag.
|
||||
* Attempt to match an endpoint based on the origin process name /
|
||||
* mask / tag.
|
||||
*/
|
||||
|
||||
ORTE_DECLSPEC orte_iof_base_endpoint_t* orte_iof_base_endpoint_match(
|
||||
const orte_process_name_t* dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
int dst_tag);
|
||||
const orte_process_name_t* target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
int target_tag);
|
||||
|
||||
/**
|
||||
* Forward the specified message out the endpoint.
|
||||
@ -142,20 +169,20 @@ ORTE_DECLSPEC orte_iof_base_endpoint_t* orte_iof_base_endpoint_match(
|
||||
|
||||
ORTE_DECLSPEC int orte_iof_base_endpoint_forward(
|
||||
orte_iof_base_endpoint_t* endpoint,
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_msg_header_t* hdr,
|
||||
const unsigned char* data);
|
||||
|
||||
/*
|
||||
* Callback when peer has closed endpoint.
|
||||
* Close the file descriptor associated with an endpoint and perform
|
||||
* any necessary cleanup.
|
||||
*/
|
||||
|
||||
ORTE_DECLSPEC void orte_iof_base_endpoint_closed(
|
||||
orte_iof_base_endpoint_t* endpoint);
|
||||
|
||||
/**
|
||||
* Callback when the specified sequence has been
|
||||
* acknowledged.
|
||||
* Callback when the next set of bytes has been acknowledged.
|
||||
*/
|
||||
|
||||
ORTE_DECLSPEC int orte_iof_base_endpoint_ack(
|
||||
@ -163,18 +190,30 @@ ORTE_DECLSPEC int orte_iof_base_endpoint_ack(
|
||||
uint32_t seq);
|
||||
|
||||
/**
|
||||
* Check for pending I/O
|
||||
* Simple check for whether we have any frags "in flight".
|
||||
*
|
||||
* Return "true" for SOURCEs if source_frags is not empty, indicating
|
||||
* that there are frags in-flight via the RML.
|
||||
*
|
||||
* Return "true" for SINKs if sink_frags is not empty, indicating that
|
||||
* there are pending frags for the fd that are either partially
|
||||
* written or have not yet been written (because writing to the fd
|
||||
* would have blocked).
|
||||
*/
|
||||
bool orte_iof_base_endpoint_have_pending_frags(
|
||||
orte_iof_base_endpoint_t* endpoint);
|
||||
|
||||
static inline bool orte_iof_base_endpoint_pending(
|
||||
orte_iof_base_endpoint_t* endpoint)
|
||||
{
|
||||
return opal_list_get_size(&endpoint->ep_frags) || (endpoint->ep_seq != endpoint->ep_ack);
|
||||
}
|
||||
/**
|
||||
* Simple check for whether we have all the ACKs that we expect.
|
||||
*
|
||||
* Return "true" for SOURCEs if ep_seq == ep_ack.
|
||||
*
|
||||
* Return "true" for SINKs always; SINK endpoints don't receive ACKs.
|
||||
*/
|
||||
bool orte_iof_base_endpoint_have_pending_acks(
|
||||
orte_iof_base_endpoint_t* endpoint);
|
||||
|
||||
#if defined(c_plusplus) || defined(__cplusplus)
|
||||
}
|
||||
#endif
|
||||
END_C_DECLS
|
||||
|
||||
#endif
|
||||
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -66,6 +67,7 @@ int orte_iof_base_flush(void)
|
||||
size_t pending;
|
||||
static int32_t lock = 0;
|
||||
|
||||
opal_output(orte_iof_base.iof_output, "CALLING IOF BASE FLUSH!");
|
||||
if(OPAL_THREAD_ADD32(&lock,1) > 1) {
|
||||
OPAL_THREAD_ADD32(&lock,-1);
|
||||
return ORTE_SUCCESS;
|
||||
@ -78,6 +80,8 @@ int orte_iof_base_flush(void)
|
||||
* wait on a timer callback to be called out of the event loop
|
||||
*/
|
||||
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"IOF BASE FLUSH: tweaking all endpoints once");
|
||||
if(opal_event_progress_thread() == false) {
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
opal_evtimer_set(&ev, orte_iof_base_timer_cb, &flushed);
|
||||
@ -89,30 +93,38 @@ int orte_iof_base_flush(void)
|
||||
opal_event_loop(OPAL_EVLOOP_NONBLOCK);
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
}
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"IOF BASE FLUSH: done tweaking all endpoints once");
|
||||
orte_iof_base.iof_waiting++;
|
||||
|
||||
/* wait for all of the endpoints to reach an idle state */
|
||||
pending = opal_list_get_size(&orte_iof_base.iof_endpoints);
|
||||
while(pending > 0) {
|
||||
do {
|
||||
pending = 0;
|
||||
for(item = opal_list_get_first(&orte_iof_base.iof_endpoints);
|
||||
item != opal_list_get_end(&orte_iof_base.iof_endpoints);
|
||||
item = opal_list_get_next(item)) {
|
||||
orte_iof_base_endpoint_t* endpoint = (orte_iof_base_endpoint_t*)item;
|
||||
if(orte_iof_base_endpoint_pending(endpoint)) {
|
||||
pending++;
|
||||
/* Count how many endpoints have fragments pending to be
|
||||
written */
|
||||
for (item = opal_list_get_first(&orte_iof_base.iof_endpoints);
|
||||
item != opal_list_get_end(&orte_iof_base.iof_endpoints);
|
||||
item = opal_list_get_next(item)) {
|
||||
orte_iof_base_endpoint_t* endpoint =
|
||||
(orte_iof_base_endpoint_t*)item;
|
||||
if (orte_iof_base_endpoint_have_pending_frags(endpoint)) {
|
||||
++pending;
|
||||
}
|
||||
}
|
||||
if(pending != 0) {
|
||||
if(opal_event_progress_thread() == false) {
|
||||
opal_condition_wait(&orte_iof_base.iof_condition, &orte_iof_base.iof_lock);
|
||||
/* If there were any with pending writes, try to make some
|
||||
progress */
|
||||
if (pending > 0) {
|
||||
if (!opal_event_progress_thread()) {
|
||||
opal_condition_wait(&orte_iof_base.iof_condition,
|
||||
&orte_iof_base.iof_lock);
|
||||
} else {
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
opal_event_loop(OPAL_EVLOOP_ONCE);
|
||||
OPAL_THREAD_LOCK(&orte_iof_base.iof_lock);
|
||||
}
|
||||
}
|
||||
}
|
||||
} while (pending > 0);
|
||||
opal_output(orte_iof_base.iof_output, "IOF BASE FLUSH: done waiting");
|
||||
orte_iof_base.iof_waiting--;
|
||||
OPAL_THREAD_UNLOCK(&orte_iof_base.iof_lock);
|
||||
OPAL_THREAD_ADD32(&lock,-1);
|
||||
|
||||
@ -90,12 +90,14 @@ static void orte_iof_base_frag_send_cb(
|
||||
*
|
||||
*/
|
||||
|
||||
int _orte_iof_base_frag_ack(orte_iof_base_frag_t* frag, const char* file, int line)
|
||||
int _orte_iof_base_frag_ack(orte_iof_base_frag_t* frag, bool do_close,
|
||||
const char* file, int line)
|
||||
{
|
||||
int rc = ORTE_SUCCESS;
|
||||
|
||||
if(frag->frag_hdr.hdr_msg.msg_len > 0) {
|
||||
frag->frag_hdr.hdr_common.hdr_type = ORTE_IOF_BASE_HDR_ACK;
|
||||
frag->frag_hdr.hdr_common.hdr_type =
|
||||
do_close ? ORTE_IOF_BASE_HDR_CLOSE : ORTE_IOF_BASE_HDR_ACK;
|
||||
ORTE_IOF_BASE_HDR_MSG_HTON(frag->frag_hdr.hdr_msg);
|
||||
|
||||
/* start non-blocking OOB call to forward header */
|
||||
|
||||
@ -74,8 +74,8 @@ ORTE_DECLSPEC OBJ_CLASS_DECLARATION(orte_iof_base_frag_t);
|
||||
* Send an acknowledgment to the peer that this fragment has been received.
|
||||
*/
|
||||
|
||||
#define orte_iof_base_frag_ack(frag) _orte_iof_base_frag_ack(frag,__FILE__,__LINE__)
|
||||
int _orte_iof_base_frag_ack(orte_iof_base_frag_t*, const char*, int);
|
||||
#define orte_iof_base_frag_ack(frag, do_close) _orte_iof_base_frag_ack((frag), (do_close), __FILE__,__LINE__)
|
||||
int _orte_iof_base_frag_ack(orte_iof_base_frag_t*, bool do_close, const char*, int);
|
||||
|
||||
#if defined(c_plusplus) || defined(__cplusplus)
|
||||
}
|
||||
|
||||
@ -1,3 +1,22 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* Copyright (c) 2004-2006 The University of Tennessee and The University
|
||||
* of Tennessee Research Foundation. All rights
|
||||
* reserved.
|
||||
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#ifndef _IOF_BASE_HEADER_
|
||||
#define _IOF_BASE_HEADER_
|
||||
|
||||
@ -10,13 +29,14 @@
|
||||
#ifdef HAVE_NETINET_IN_H
|
||||
#include <netinet/in.h>
|
||||
#endif
|
||||
|
||||
|
||||
#define ORTE_IOF_BASE_HDR_MSG 0
|
||||
#define ORTE_IOF_BASE_HDR_ACK 1
|
||||
#define ORTE_IOF_BASE_HDR_PUB 2
|
||||
#define ORTE_IOF_BASE_HDR_UNPUB 3
|
||||
#define ORTE_IOF_BASE_HDR_SUB 4
|
||||
#define ORTE_IOF_BASE_HDR_UNSUB 5
|
||||
#define ORTE_IOF_BASE_HDR_CLOSE 6
|
||||
|
||||
|
||||
/*
|
||||
@ -48,7 +68,7 @@ typedef struct orte_iof_base_common_header_t orte_iof_base_common_header_t;
|
||||
|
||||
struct orte_iof_base_msg_header_t {
|
||||
orte_iof_base_common_header_t hdr_common;
|
||||
orte_process_name_t msg_src;
|
||||
orte_process_name_t msg_origin;
|
||||
orte_process_name_t msg_proxy;
|
||||
int32_t msg_tag;
|
||||
uint32_t msg_seq;
|
||||
@ -58,7 +78,7 @@ typedef struct orte_iof_base_msg_header_t orte_iof_base_msg_header_t;
|
||||
|
||||
#define ORTE_IOF_BASE_HDR_MSG_NTOH(h) \
|
||||
ORTE_IOF_BASE_HDR_CMN_NTOH((h).hdr_common); \
|
||||
ORTE_PROCESS_NAME_NTOH((h).msg_src); \
|
||||
ORTE_PROCESS_NAME_NTOH((h).msg_origin); \
|
||||
ORTE_PROCESS_NAME_NTOH((h).msg_proxy); \
|
||||
(h).msg_tag = ntohl((h).msg_tag); \
|
||||
(h).msg_seq = ntohl((h).msg_seq); \
|
||||
@ -66,7 +86,7 @@ typedef struct orte_iof_base_msg_header_t orte_iof_base_msg_header_t;
|
||||
|
||||
#define ORTE_IOF_BASE_HDR_MSG_HTON(h) \
|
||||
ORTE_IOF_BASE_HDR_CMN_HTON((h).hdr_common); \
|
||||
ORTE_PROCESS_NAME_HTON((h).msg_src); \
|
||||
ORTE_PROCESS_NAME_HTON((h).msg_origin); \
|
||||
ORTE_PROCESS_NAME_HTON((h).msg_proxy); \
|
||||
(h).msg_tag = htonl((h).msg_tag); \
|
||||
(h).msg_seq = htonl((h).msg_seq); \
|
||||
@ -106,28 +126,28 @@ typedef struct orte_iof_base_pub_header_t orte_iof_base_pub_header_t;
|
||||
|
||||
struct orte_iof_base_sub_header_t {
|
||||
orte_iof_base_common_header_t hdr_common;
|
||||
orte_process_name_t src_name;
|
||||
orte_ns_cmp_bitmask_t src_mask;
|
||||
int32_t src_tag;
|
||||
orte_process_name_t dst_name;
|
||||
orte_ns_cmp_bitmask_t dst_mask;
|
||||
int32_t dst_tag;
|
||||
orte_process_name_t origin_name;
|
||||
orte_ns_cmp_bitmask_t origin_mask;
|
||||
int32_t origin_tag;
|
||||
orte_process_name_t target_name;
|
||||
orte_ns_cmp_bitmask_t target_mask;
|
||||
int32_t target_tag;
|
||||
};
|
||||
typedef struct orte_iof_base_sub_header_t orte_iof_base_sub_header_t;
|
||||
|
||||
#define ORTE_IOF_BASE_HDR_SUB_NTOH(h) \
|
||||
ORTE_IOF_BASE_HDR_CMN_NTOH((h).hdr_common); \
|
||||
ORTE_PROCESS_NAME_NTOH((h).src_name); \
|
||||
(h).src_tag = ntohl((h).src_tag); \
|
||||
ORTE_PROCESS_NAME_NTOH((h).dst_name); \
|
||||
(h).dst_tag = ntohl((h).dst_tag);
|
||||
ORTE_PROCESS_NAME_NTOH((h).origin_name); \
|
||||
(h).origin_tag = ntohl((h).origin_tag); \
|
||||
ORTE_PROCESS_NAME_NTOH((h).target_name); \
|
||||
(h).target_tag = ntohl((h).target_tag);
|
||||
|
||||
#define ORTE_IOF_BASE_HDR_SUB_HTON(h) \
|
||||
ORTE_IOF_BASE_HDR_CMN_HTON((h).hdr_common); \
|
||||
ORTE_PROCESS_NAME_HTON((h).src_name); \
|
||||
(h).src_tag = htonl((h).src_tag); \
|
||||
ORTE_PROCESS_NAME_HTON((h).dst_name); \
|
||||
(h).dst_tag = htonl((h).dst_tag);
|
||||
ORTE_PROCESS_NAME_HTON((h).origin_name); \
|
||||
(h).origin_tag = htonl((h).origin_tag); \
|
||||
ORTE_PROCESS_NAME_HTON((h).target_name); \
|
||||
(h).target_tag = htonl((h).target_tag);
|
||||
|
||||
/**
|
||||
* Union of all header types.
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
/*
|
||||
* Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana
|
||||
* Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
|
||||
* University Research and Technology
|
||||
* Corporation. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The University of Tennessee and The University
|
||||
@ -91,7 +91,7 @@ int orte_iof_base_open(void)
|
||||
OBJ_CLASS(orte_iof_base_frag_t),
|
||||
0, /* number to initially allocate */
|
||||
-1, /* maximum elements to allocate */
|
||||
32 ); /* number per allocation */
|
||||
16 ); /* number per allocation */
|
||||
|
||||
/* Open up all available components */
|
||||
if (ORTE_SUCCESS !=
|
||||
|
||||
@ -26,7 +26,7 @@
|
||||
* used with any file descriptor stream.
|
||||
*
|
||||
* In practice, the IOF acts as a multiplexor between local file
|
||||
* descriptors and the OOB; the OOB relays information from local file
|
||||
* descriptors and the RML; the RML relays information from local file
|
||||
* descriptors to remote file descriptors. Note that the IOF allows
|
||||
* many-to-one mappings; SOURCE streams can be directed to multiple
|
||||
* destinations and SINK streams can receive input from multiple
|
||||
@ -67,6 +67,9 @@
|
||||
* if a stdout descriptor is available from process X, then process X
|
||||
* needs to publish it (and make it a stream) in order to make that
|
||||
* stdout stream available to any other process.
|
||||
* --> today, this isn't necessarily true for the proxy because
|
||||
* everything is atuomatically sent to the svc. But the proxy
|
||||
* should be fixed someday to make this definition consistent.
|
||||
*
|
||||
* unpublish: The opposite of publish; when a stream is unpublished,
|
||||
* the content from that file desciptor is no longer available to
|
||||
@ -100,6 +103,39 @@
|
||||
* flush: Block until all pending data has been written down local
|
||||
* file descriptors and/or completed sending across the OOB to remote
|
||||
* process targets.
|
||||
*
|
||||
* Two terms that are used in the IOF interface are "origin" and
|
||||
* "target" indicating the process where data started and where it is
|
||||
* going. These terms are used to distinguish IOF component
|
||||
* implementation details because data does not necessarily only from
|
||||
* the SOURCE process to the SINK process. In practice, data can flow
|
||||
* from a SINK to a SOURCE (e.g., an ACK), or be routed through a
|
||||
* proxy. So the "origin" and "target" processes are those where the
|
||||
* data started and will terminate, respectively, regardless of the
|
||||
* designation of the originating process (as the SOURCE, SINK, or
|
||||
* proxy) and the destination process (as the SOURCE, SINK, or proxy).
|
||||
*
|
||||
* Additionally, the "proxy" is as it is described above: it may be
|
||||
* the origin or target itself, or it may be an intermediary acting on
|
||||
* behalf of the origin or target.
|
||||
*
|
||||
* Examples:
|
||||
*
|
||||
* 1. mpirun -np 1 hostname
|
||||
* Assume that orteds and an HNP are used. An orted will be
|
||||
* launched on the same node as "hostname". It will act as a proxy
|
||||
* for the hostname process' stdin, stdout, and stderr. Data read
|
||||
* by the orted from the hostname process stdout will be sent to
|
||||
* mpirun. In this case, the hostname process is the origin,
|
||||
* mpirun is the target, and the orted is the proxy.
|
||||
*
|
||||
* 2. mpirun -np 1 read_stdin < input_filename
|
||||
* Assume that orteds and an HNP are used. As with #1, an orted
|
||||
* will proxy the stdin, stdout, and stderr of the read_stdin
|
||||
* process. When mpirun reads data on its stdin, it will forward
|
||||
* it to the orted to write down the pipe to the read_stdin
|
||||
* process. In this case, mpirun is both the origin process *and*
|
||||
* proxy, and read_stdin is the target.
|
||||
*/
|
||||
|
||||
#ifndef ORTE_IOF_H
|
||||
@ -113,11 +149,9 @@
|
||||
#include "opal/mca/crs/crs.h"
|
||||
#include "opal/mca/crs/base/base.h"
|
||||
|
||||
#if defined(c_plusplus) || defined(__cplusplus)
|
||||
extern "C" {
|
||||
#endif
|
||||
BEGIN_C_DECLS
|
||||
|
||||
/* suggested tag values */
|
||||
/* Predefined tag values */
|
||||
enum {
|
||||
ORTE_IOF_ANY = -1,
|
||||
ORTE_IOF_STDIN = 0,
|
||||
@ -136,137 +170,152 @@ typedef int orte_iof_base_mode_t;
|
||||
|
||||
/**
|
||||
* Publish a local file descriptor as an endpoint that is logically
|
||||
* associated with the specified process name (e.g. file descriptor
|
||||
* corresponding to the master side of a pipe/pty connected to a
|
||||
* child process)
|
||||
* associated with the specified origin process name. The file
|
||||
* descriptor may be local to this process (in which case the origin
|
||||
* process name is this process' name), or it may be a pipe to another
|
||||
* process (i.e., this process is acting as a proxy for another
|
||||
* process -- typically the case for stdin, stdout, stderr).
|
||||
*
|
||||
* @param name Process name associated with the endpoint.
|
||||
* @param mode Is the endpoint an input or output.
|
||||
* @param origin Origin process name associated with the endpoint (not
|
||||
* the proxy process).
|
||||
* @param mode Is the endpoint an input or output (SOURCE or SINK)
|
||||
* @param tag The logical tag associated with this file descriptor.
|
||||
* @param fd Local file descriptor
|
||||
*
|
||||
*/
|
||||
|
||||
typedef int (*orte_iof_base_publish_fn_t)(
|
||||
const orte_process_name_t* name,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_mode_t mode,
|
||||
orte_iof_base_tag_t tag,
|
||||
int fd
|
||||
);
|
||||
|
||||
/**
|
||||
* Remove all endpoints matching the specified process
|
||||
* name, mask and tag values.
|
||||
* Remove all endpoints matching the specified origin process name,
|
||||
* mask and tag values.
|
||||
*
|
||||
* @param name Process name associated with the endpoint.
|
||||
* @param name Origin process name associated with the endpoint.
|
||||
* @param mask A mask indicating the set of processes to unpublish.
|
||||
* @param tag The endpoint tag.
|
||||
*
|
||||
*/
|
||||
|
||||
typedef int (*orte_iof_base_unpublish_fn_t)(
|
||||
const orte_process_name_t* name,
|
||||
const orte_process_name_t* origin,
|
||||
orte_ns_cmp_bitmask_t mask,
|
||||
orte_iof_base_tag_t tag
|
||||
);
|
||||
|
||||
/**
|
||||
* Explicitly push data from the specified input file
|
||||
* descriptor to the indicated set of peers.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor for input.
|
||||
* Callback function for subscriptions (see orte_iof_base_subscribe_fn_t).
|
||||
*/
|
||||
|
||||
typedef int (*orte_iof_base_push_fn_t)(
|
||||
const orte_process_name_t* dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
orte_iof_base_tag_t dst_tag,
|
||||
int fd
|
||||
);
|
||||
|
||||
/**
|
||||
* Explicitly pull data from the specified set of peers
|
||||
* and dump to the indicated output file descriptor.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor for output.
|
||||
*/
|
||||
|
||||
typedef int (*orte_iof_base_pull_fn_t)(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
int fd
|
||||
);
|
||||
|
||||
/**
|
||||
* Setup buffering for a specified set of endpoints.
|
||||
*/
|
||||
|
||||
typedef int (*orte_iof_base_buffer_fn_t)(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
size_t buffer_size
|
||||
);
|
||||
|
||||
/*
|
||||
* Subscribe to receive a callback on receipt of data
|
||||
* from a specified set of peers.
|
||||
*/
|
||||
|
||||
typedef void (*orte_iof_base_callback_fn_t)(
|
||||
orte_process_name_t* src_name,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
orte_process_name_t* origin_name,
|
||||
orte_iof_base_tag_t orign_tag,
|
||||
void *cbdata,
|
||||
const unsigned char* data,
|
||||
size_t count
|
||||
);
|
||||
|
||||
/**
|
||||
* Subscribe to receive a callback on receipt of data from a specified
|
||||
* set of origin peers.
|
||||
*
|
||||
* This function is a general purpose utility for getting data from a
|
||||
* stream; the incoming fragment is delivered to the callback in a
|
||||
* buffer. You can do whatever you want with the buffer when you get
|
||||
* the callback (e.g., buffer it, call syslog, ...etc.).
|
||||
*
|
||||
* Note that the orte_iof_base_pull_fn_t is a customized common-case
|
||||
* version of this function; it always takes incoming fragments from a
|
||||
* stream and writes them down an fd.
|
||||
*/
|
||||
typedef int (*orte_iof_base_subscribe_fn_t)(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag,
|
||||
orte_iof_base_callback_fn_t cb,
|
||||
void* cbdata
|
||||
);
|
||||
|
||||
/**
|
||||
* Delete a subscription created by orte_iof_base_subscribe_fn_t.
|
||||
*/
|
||||
typedef int (*orte_iof_base_unsubscribe_fn_t)(
|
||||
const orte_process_name_t* src_name,
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag
|
||||
);
|
||||
|
||||
|
||||
/*
|
||||
/**
|
||||
* Explicitly push data from the specified input file descriptor to
|
||||
* the indicated set of SINK peers.
|
||||
*
|
||||
* This function is a shortcut for publishing a SOURCE stream and
|
||||
* tying that stream to an fd that is providing data to be sent across
|
||||
* the stream (e.g., read from stdin and push it out to a stream).
|
||||
* Any data that appears on the fd will automatically be read and sent
|
||||
* across the stream.
|
||||
*
|
||||
* @param sink_name Name used to qualify set of target peers.
|
||||
* @param sink_mask Mask that specified how name is interpreted.
|
||||
* @param sink_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor for input.
|
||||
*/
|
||||
typedef int (*orte_iof_base_push_fn_t)(
|
||||
const orte_process_name_t* sink_name,
|
||||
orte_ns_cmp_bitmask_t sink_mask,
|
||||
orte_iof_base_tag_t sink_tag,
|
||||
int fd
|
||||
);
|
||||
|
||||
/**
|
||||
* Explicitly pull data from the specified set of SOURCE peers and
|
||||
* dump to the indicated output file descriptor.
|
||||
*
|
||||
* This function is a shortcut for subscribing to a SOURCE stream and
|
||||
* tying that stream to an fd that will consume the data received from
|
||||
* the stream (i.e., get a fragment from a stream and write it down an
|
||||
* fd). Any fragments that arrive on the stream will automatically be
|
||||
* written down the fd.
|
||||
*
|
||||
* @param source_name Name used to qualify set of origin peers.
|
||||
* @param source_mask Mask that specified how name is interpreted.
|
||||
* @param source_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor for output.
|
||||
*/
|
||||
typedef int (*orte_iof_base_pull_fn_t)(
|
||||
const orte_process_name_t* source_name,
|
||||
orte_ns_cmp_bitmask_t source_mask,
|
||||
orte_iof_base_tag_t source_tag,
|
||||
int fd
|
||||
);
|
||||
|
||||
/**
|
||||
* Flush all output and block until output is delivered.
|
||||
*/
|
||||
|
||||
typedef int (*orte_iof_base_flush_fn_t)(void);
|
||||
|
||||
/**
|
||||
* Shut down an IOF module
|
||||
*/
|
||||
typedef int (*orte_iof_base_finalize_fn_t)(void);
|
||||
|
||||
/*
|
||||
* FT Event Notification
|
||||
*/
|
||||
typedef int (*orte_iof_base_ft_event_fn_t)(int state);
|
||||
/**
|
||||
* FT Event Notification
|
||||
*/
|
||||
typedef int (*orte_iof_base_ft_event_fn_t)(int state);
|
||||
|
||||
/**
|
||||
* IOF module.
|
||||
*/
|
||||
|
||||
struct orte_iof_base_module_1_0_0_t {
|
||||
orte_iof_base_publish_fn_t iof_publish;
|
||||
orte_iof_base_unpublish_fn_t iof_unpublish;
|
||||
orte_iof_base_push_fn_t iof_push;
|
||||
orte_iof_base_pull_fn_t iof_pull;
|
||||
orte_iof_base_subscribe_fn_t iof_subscribe;
|
||||
orte_iof_base_unsubscribe_fn_t iof_unsubscribe;
|
||||
orte_iof_base_push_fn_t iof_push;
|
||||
orte_iof_base_pull_fn_t iof_pull;
|
||||
orte_iof_base_flush_fn_t iof_flush;
|
||||
orte_iof_base_finalize_fn_t iof_finalize;
|
||||
orte_iof_base_ft_event_fn_t ft_event;
|
||||
@ -277,10 +326,9 @@ typedef orte_iof_base_module_1_0_0_t orte_iof_base_module_t;
|
||||
ORTE_DECLSPEC extern orte_iof_base_module_t orte_iof;
|
||||
|
||||
/**
|
||||
* IOF component descriptor. Contains component version information
|
||||
* and component open/close/init functions.
|
||||
* IOF component init function. Contains component version
|
||||
* information and component open/close/init functions.
|
||||
*/
|
||||
|
||||
typedef orte_iof_base_module_t* (*orte_iof_base_component_init_fn_t)(
|
||||
int *priority,
|
||||
bool *allow_user_threads,
|
||||
@ -294,7 +342,9 @@ struct orte_iof_base_component_1_0_0_t {
|
||||
};
|
||||
typedef struct orte_iof_base_component_1_0_0_t orte_iof_base_component_1_0_0_t;
|
||||
typedef struct orte_iof_base_component_1_0_0_t orte_iof_base_component_t;
|
||||
|
||||
|
||||
END_C_DECLS
|
||||
|
||||
/*
|
||||
* Macro for use in components that are of type iof v1.0.0
|
||||
*/
|
||||
@ -304,7 +354,4 @@ typedef struct orte_iof_base_component_1_0_0_t orte_iof_base_component_t;
|
||||
/* iof v1.0 */ \
|
||||
"iof", 1, 0, 0
|
||||
|
||||
#if defined(c_plusplus) || defined(__cplusplus)
|
||||
}
|
||||
#endif
|
||||
#endif /* ORTE_IOF_H */
|
||||
|
||||
@ -9,13 +9,16 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
|
||||
#include "orte_config.h"
|
||||
|
||||
#include <errno.h>
|
||||
#ifdef HAVE_UNISTD_H
|
||||
#include <unistd.h>
|
||||
@ -40,89 +43,105 @@
|
||||
orte_iof_base_module_t orte_iof_proxy_module = {
|
||||
orte_iof_proxy_publish,
|
||||
orte_iof_proxy_unpublish,
|
||||
orte_iof_proxy_push,
|
||||
orte_iof_proxy_pull,
|
||||
orte_iof_proxy_subscribe,
|
||||
orte_iof_proxy_unsubscribe,
|
||||
orte_iof_proxy_push,
|
||||
orte_iof_proxy_pull,
|
||||
orte_iof_base_flush,
|
||||
orte_iof_proxy_finalize,
|
||||
orte_iof_proxy_ft_event
|
||||
};
|
||||
|
||||
int orte_iof_proxy_finalize(void ) {
|
||||
/*
|
||||
* Finalize module; nothing to do
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_finalize(void )
|
||||
{
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Publish a local file descriptor as an endpoint that is logically
|
||||
* associated with the specified process name (e.g. master side of a
|
||||
* pipe/pty connected to a child process)
|
||||
*
|
||||
* @param name
|
||||
* @param mode
|
||||
* @param tag
|
||||
* @param fd
|
||||
* Create an endpoint for a local file descriptor and "publish" it
|
||||
* under the name of the origin process. If the publish mode is a
|
||||
* SINK, then create a publication entry for it so that incoming
|
||||
* messages can be forwarded to it.
|
||||
*
|
||||
* SOURCEs do not need to create publication records because a) the
|
||||
* endpoint will automatically wake up the event engine and read off
|
||||
* the fd whenever there is data available, and b) this data is then
|
||||
* automatically sent to the iof svc component for possible
|
||||
* forwarding.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_publish(
|
||||
const orte_process_name_t* name,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_mode_t mode,
|
||||
orte_iof_base_tag_t tag,
|
||||
int fd)
|
||||
{
|
||||
int rc;
|
||||
|
||||
if(mca_iof_proxy_component.proxy_debug > 1) {
|
||||
if (orte_iof_base.iof_output >= 0) {
|
||||
char* name_str;
|
||||
orte_ns.get_proc_name_string(&name_str, name);
|
||||
opal_output(0, "orte_iof_proxy_publish(%s,%d,%d,%d)\n", name_str, mode, tag, fd);
|
||||
orte_ns.get_proc_name_string(&name_str, origin);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_proxy_publish(%s,%d,%d,%d)\n",
|
||||
name_str, mode, tag, fd);
|
||||
free(name_str);
|
||||
}
|
||||
|
||||
/* publish to server */
|
||||
if(mode == ORTE_IOF_SINK) {
|
||||
rc = orte_iof_proxy_svc_publish(name,tag);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* setup a local endpoint to reflect registration */
|
||||
rc = orte_iof_base_endpoint_create(
|
||||
name,
|
||||
origin,
|
||||
mode,
|
||||
tag,
|
||||
fd);
|
||||
return rc;
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* publish to server */
|
||||
if (ORTE_IOF_SINK == mode) {
|
||||
rc = orte_iof_proxy_svc_publish(origin, tag);
|
||||
if (rc != ORTE_SUCCESS) {
|
||||
return rc;
|
||||
}
|
||||
}
|
||||
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Remove all registrations matching the specified process
|
||||
* Remove all registrations matching the specified origin process
|
||||
* name, mask and tag values.
|
||||
*
|
||||
* @param name
|
||||
* @param mask
|
||||
* @param tag
|
||||
*
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_unpublish(
|
||||
const orte_process_name_t* name,
|
||||
const orte_process_name_t* origin,
|
||||
orte_ns_cmp_bitmask_t mask,
|
||||
orte_iof_base_tag_t tag)
|
||||
{
|
||||
int rc;
|
||||
|
||||
#if 0
|
||||
{
|
||||
int i = 0;
|
||||
opal_output(orte_iof_base.iof_output, "[%lu,%lu,%lu] orted: ******** ABOUT TO IOF PROXY UNPUBLISH, %d", ORTE_NAME_ARGS(orte_process_info.my_name), getpid());
|
||||
fflush(stderr);
|
||||
while (0 == i) sleep(5);
|
||||
}
|
||||
#endif
|
||||
|
||||
/* cleanup server */
|
||||
orte_iof_proxy_svc_unpublish(
|
||||
name,
|
||||
origin,
|
||||
mask,
|
||||
tag);
|
||||
|
||||
/* setup a local endpoint to reflect registration */
|
||||
/* delete local endpoint */
|
||||
rc = orte_iof_base_endpoint_delete(
|
||||
name,
|
||||
origin,
|
||||
mask,
|
||||
tag);
|
||||
return rc;
|
||||
@ -131,59 +150,51 @@ int orte_iof_proxy_unpublish(
|
||||
|
||||
/**
|
||||
* Explicitly push data from the specified file descriptor
|
||||
* to the indicated set of peers.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor.
|
||||
* to the indicated SINK set of peers.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_push(
|
||||
const orte_process_name_t* dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
orte_iof_base_tag_t dst_tag,
|
||||
const orte_process_name_t* sink_name,
|
||||
orte_ns_cmp_bitmask_t sink_mask,
|
||||
orte_iof_base_tag_t sink_tag,
|
||||
int fd)
|
||||
{
|
||||
int rc;
|
||||
|
||||
/* send a subscription to server on behalf of the destination */
|
||||
rc = orte_iof_proxy_svc_subscribe(
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
dst_tag,
|
||||
dst_name,
|
||||
dst_mask,
|
||||
dst_tag
|
||||
);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
return rc;
|
||||
|
||||
/* setup a local endpoint to reflect registration */
|
||||
/* setup a local endpoint to reflect registration. Do this before
|
||||
we send the subscription to the server in case a callback
|
||||
occurs *while* we are sending the subscription request. */
|
||||
rc = orte_iof_base_endpoint_create(
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_IOF_SOURCE,
|
||||
dst_tag,
|
||||
sink_tag,
|
||||
fd);
|
||||
|
||||
return rc;
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* send a subscription to server on behalf of the destination */
|
||||
rc = orte_iof_proxy_svc_subscribe(
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
sink_tag,
|
||||
sink_name,
|
||||
sink_mask,
|
||||
sink_tag
|
||||
);
|
||||
return rc;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Explicitly pull data from the specified set of peers
|
||||
* and dump to the indicated file descriptor.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor.
|
||||
* Explicitly pull data from the specified set of SOURCE peers and
|
||||
* dump to the indicated file descriptor.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_pull(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t* source_name,
|
||||
orte_ns_cmp_bitmask_t source_mask,
|
||||
orte_iof_base_tag_t source_tag,
|
||||
int fd)
|
||||
{
|
||||
/* setup a local endpoint */
|
||||
@ -191,9 +202,9 @@ int orte_iof_proxy_pull(
|
||||
rc = orte_iof_base_endpoint_create(
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_IOF_SINK,
|
||||
src_tag,
|
||||
source_tag,
|
||||
fd);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
return rc;
|
||||
}
|
||||
@ -201,21 +212,21 @@ int orte_iof_proxy_pull(
|
||||
/* publish this endpoint */
|
||||
rc = orte_iof_proxy_svc_publish(
|
||||
ORTE_PROC_MY_NAME,
|
||||
src_tag);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
source_tag);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* subscribe to peer */
|
||||
rc = orte_iof_proxy_svc_subscribe(
|
||||
src_name,
|
||||
src_mask,
|
||||
src_tag,
|
||||
source_name,
|
||||
source_mask,
|
||||
source_tag,
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
src_tag);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
source_tag);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
return rc;
|
||||
}
|
||||
@ -223,72 +234,68 @@ int orte_iof_proxy_pull(
|
||||
return rc;
|
||||
}
|
||||
|
||||
/**
|
||||
* Setup buffering for a specified set of endpoints.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_buffer(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
size_t buffer_size)
|
||||
{
|
||||
return ORTE_ERROR;
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* Subscribe to receive a callback on receipt of data
|
||||
* from a specified set of peers.
|
||||
* from a specified set of origin peers.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_subscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag,
|
||||
orte_iof_base_callback_fn_t cbfunc,
|
||||
void* cbdata)
|
||||
{
|
||||
int rc;
|
||||
|
||||
/* create a local registration to reflect the callback */
|
||||
rc = orte_iof_base_callback_create(ORTE_PROC_MY_NAME,src_tag,cbfunc,cbdata);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
rc = orte_iof_base_callback_create(ORTE_PROC_MY_NAME,origin_tag,cbfunc,cbdata);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* send a subscription message to the service */
|
||||
rc = orte_iof_proxy_svc_subscribe(
|
||||
src_name,
|
||||
src_mask,
|
||||
src_tag,
|
||||
origin_name,
|
||||
origin_mask,
|
||||
origin_tag,
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
src_tag);
|
||||
origin_tag);
|
||||
return rc;
|
||||
}
|
||||
|
||||
/*
|
||||
* Remove a subscription
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_unsubscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag)
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag)
|
||||
{
|
||||
int rc;
|
||||
|
||||
/* send an unsubscribe message to the service */
|
||||
rc = orte_iof_proxy_svc_unsubscribe(
|
||||
src_name,
|
||||
src_mask,
|
||||
src_tag,
|
||||
origin_name,
|
||||
origin_mask,
|
||||
origin_tag,
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
src_tag);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
origin_tag);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* remove local callback */
|
||||
return orte_iof_base_callback_delete(ORTE_PROC_MY_NAME,src_tag);
|
||||
return orte_iof_base_callback_delete(ORTE_PROC_MY_NAME,origin_tag);
|
||||
}
|
||||
|
||||
/*
|
||||
* FT event
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_ft_event(int state) {
|
||||
int ret, exit_status = ORTE_SUCCESS;
|
||||
|
||||
|
||||
@ -9,6 +9,8 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* Copyright (c) 2007 Sun Microsystems, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -17,6 +19,45 @@
|
||||
*/
|
||||
/**
|
||||
* @file
|
||||
*
|
||||
* The proxy IOF component is used in non-HNP processes. It is used
|
||||
* to proxy all IOF actions back to the "svc" IOF component (i.e., the
|
||||
* IOF component that runs in the HNP). The proxy IOF component is
|
||||
* typically loaded in an orted and then tied to the stdin, stdout,
|
||||
* and stderr streams of created child processes via pipes. The proxy
|
||||
* IOF component in the orted then acts as the delay between the
|
||||
* stdin/stdout/stderr pipes and the svc IOF component in the HNP.
|
||||
* This design allows us to manipulate stdin/stdout/stderr from before
|
||||
* main() in the child process.
|
||||
*
|
||||
* Publish actions for SINKs are pushed back to the svc/HNP. Publish
|
||||
* actions for SOURCEs are not pushed back to SINKs because all data
|
||||
* fragments from SOURCEs are automatically sent back to the svc/HNP.
|
||||
*
|
||||
* All unpublish actions are pushed back to the svc/HNP (I'm not sure
|
||||
* why -- perhaps this is a bug?).
|
||||
*
|
||||
* Push and pull actions are essentially implemented in terms of
|
||||
* subscribe / unsubscribe.
|
||||
*
|
||||
* Subscribe / unsubscribe actions are fairly straightforward.
|
||||
*
|
||||
* Much of the intelligence of this component is actually contained in
|
||||
* iof_base_endpoint.c (reading and writing to local file descriptors,
|
||||
* setting up events based on file descriptors, etc.).
|
||||
*
|
||||
* A non-blocking OOB receive is posted at the initializtion of this
|
||||
* component to receive all messages from the svc/HNP (e.g., data
|
||||
* fragments from streams, ACKs to fragments).
|
||||
*
|
||||
* Flow control is employed on a per-stream basis to ensure that
|
||||
* SOURCEs don't overwhelm SINK resources (E.g., send an entire input
|
||||
* file to an orted before the target process has read any of it).
|
||||
*
|
||||
* Important: this component is designed to work with the svc IOF
|
||||
* component only. If we ever do a different IOF implementation
|
||||
* scheme, it is likely that only some of this component will be
|
||||
* useful for cannibalisation (if any at all).
|
||||
*/
|
||||
#ifndef ORTE_IOF_PROXY_H
|
||||
#define ORTE_IOF_PROXY_H
|
||||
@ -28,15 +69,7 @@ extern "C" {
|
||||
#endif
|
||||
|
||||
/**
|
||||
* Publish a local file descriptor as an endpoint that is logically
|
||||
* associated with the specified process name (e.g. master side of a
|
||||
* pipe/pty connected to a child process)
|
||||
*
|
||||
* @param name
|
||||
* @param mode
|
||||
* @param tag
|
||||
* @param fd
|
||||
*
|
||||
* Module publish
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_publish(
|
||||
@ -47,13 +80,7 @@ int orte_iof_proxy_publish(
|
||||
);
|
||||
|
||||
/**
|
||||
* Remove all registrations matching the specified process
|
||||
* name, mask and tag values.
|
||||
*
|
||||
* @param name
|
||||
* @param mask
|
||||
* @param tag
|
||||
*
|
||||
* Module unpublish
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_unpublish(
|
||||
@ -63,13 +90,7 @@ int orte_iof_proxy_unpublish(
|
||||
);
|
||||
|
||||
/**
|
||||
* Explicitly push data from the specified file descriptor
|
||||
* to the indicated set of peers.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor.
|
||||
* Module push
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_push(
|
||||
@ -80,13 +101,7 @@ int orte_iof_proxy_push(
|
||||
);
|
||||
|
||||
/**
|
||||
* Explicitly pull data from the specified set of peers
|
||||
* and dump to the indicated file descriptor.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor.
|
||||
* Module pull
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_pull(
|
||||
@ -97,19 +112,7 @@ int orte_iof_proxy_pull(
|
||||
);
|
||||
|
||||
/**
|
||||
* Setup buffering for a specified set of endpoints.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_buffer(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
size_t buffer_size
|
||||
);
|
||||
|
||||
/*
|
||||
* Subscribe to receive a callback on receipt of data
|
||||
* from a specified set of peers.
|
||||
* Module subscribe
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_subscribe(
|
||||
@ -120,14 +123,26 @@ int orte_iof_proxy_subscribe(
|
||||
void* cbdata
|
||||
);
|
||||
|
||||
/**
|
||||
* Module unsubscribe
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_unsubscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag
|
||||
);
|
||||
|
||||
/**
|
||||
* Module finalize
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_finalize( void );
|
||||
|
||||
/**
|
||||
* Module FT event
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_ft_event(int state);
|
||||
|
||||
/**
|
||||
@ -135,7 +150,6 @@ int orte_iof_proxy_ft_event(int state);
|
||||
*/
|
||||
struct orte_iof_proxy_component_t {
|
||||
orte_iof_base_component_t super;
|
||||
int proxy_debug;
|
||||
struct iovec proxy_iov[1];
|
||||
};
|
||||
typedef struct orte_iof_proxy_component_t orte_iof_proxy_component_t;
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -71,42 +72,15 @@ orte_iof_proxy_component_t mca_iof_proxy_component = {
|
||||
},
|
||||
|
||||
orte_iof_proxy_init
|
||||
},
|
||||
false,
|
||||
/* {{NULL, 0}} - Let the compiler initialize it so it won't complain on Windows
|
||||
* where the order of the length and the pointer is different.
|
||||
*/
|
||||
}
|
||||
};
|
||||
|
||||
#if 0
|
||||
static char* orte_iof_proxy_param_register_string(
|
||||
const char* param_name,
|
||||
const char* default_value)
|
||||
{
|
||||
char *param_value;
|
||||
int id = mca_base_param_register_string("iof","proxy",param_name,NULL,default_value);
|
||||
mca_base_param_lookup_string(id, ¶m_value);
|
||||
return param_value;
|
||||
}
|
||||
#endif
|
||||
|
||||
static int orte_iof_proxy_param_register_int(
|
||||
const char* param_name,
|
||||
int default_value)
|
||||
{
|
||||
int id = mca_base_param_register_int("iof","proxy",param_name,NULL,default_value);
|
||||
int param_value = default_value;
|
||||
mca_base_param_lookup_int(id,¶m_value);
|
||||
return param_value;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* component open/close/init function
|
||||
*/
|
||||
static int orte_iof_proxy_open(void)
|
||||
{
|
||||
mca_iof_proxy_component.proxy_debug = orte_iof_proxy_param_register_int("debug",1);
|
||||
/* Nothing to do */
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
@ -122,10 +96,10 @@ orte_iof_proxy_init(int* priority, bool *allow_multi_user_threads, bool *have_hi
|
||||
*allow_multi_user_threads = true;
|
||||
*have_hidden_threads = false;
|
||||
|
||||
/* post receive with oob */
|
||||
/* post a non-blocking, persistent RML receive to get messages
|
||||
from the svc IOF component */
|
||||
mca_iof_proxy_component.proxy_iov[0].iov_base = NULL;
|
||||
mca_iof_proxy_component.proxy_iov[0].iov_len = 0;
|
||||
|
||||
rc = orte_rml.recv_nb(
|
||||
ORTE_NAME_WILDCARD,
|
||||
mca_iof_proxy_component.proxy_iov,
|
||||
@ -136,22 +110,20 @@ orte_iof_proxy_init(int* priority, bool *allow_multi_user_threads, bool *have_hi
|
||||
NULL
|
||||
);
|
||||
if(rc < 0) {
|
||||
opal_output(0, "orte_iof_proxy_init: unable to post non-blocking recv");
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_proxy_init: unable to post non-blocking recv");
|
||||
return NULL;
|
||||
}
|
||||
initialized = true;
|
||||
return &orte_iof_proxy_module;
|
||||
}
|
||||
|
||||
/**
|
||||
*
|
||||
*/
|
||||
|
||||
static int orte_iof_proxy_close(void)
|
||||
{
|
||||
int rc = ORTE_SUCCESS;
|
||||
|
||||
if (initialized) {
|
||||
/* Cancel the RML receive */
|
||||
rc = orte_rml.recv_cancel(ORTE_NAME_WILDCARD, ORTE_RML_TAG_IOF_SVC);
|
||||
}
|
||||
return rc;
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -33,18 +34,17 @@
|
||||
*/
|
||||
|
||||
static void orte_iof_proxy_svc_msg(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_msg_header_t* msg,
|
||||
unsigned char* data);
|
||||
|
||||
static void orte_iof_proxy_svc_ack(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_msg_header_t* msg);
|
||||
|
||||
|
||||
/*
|
||||
* Publish the availability of a local endpoint
|
||||
* to the servver.
|
||||
* Send a "publish" request to the svc component
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_svc_publish(
|
||||
@ -61,7 +61,7 @@ int orte_iof_proxy_svc_publish(
|
||||
hdr.hdr_pub.pub_proxy = *ORTE_PROC_MY_NAME;
|
||||
hdr.hdr_pub.pub_mask = ORTE_NS_CMP_ALL;
|
||||
hdr.hdr_pub.pub_tag = tag;
|
||||
ORTE_IOF_BASE_HDR_PUB_NTOH(hdr.hdr_pub);
|
||||
ORTE_IOF_BASE_HDR_PUB_HTON(hdr.hdr_pub);
|
||||
|
||||
iov.iov_base = (IOVBASE_TYPE*)&hdr;
|
||||
iov.iov_len = sizeof(hdr);
|
||||
@ -81,7 +81,7 @@ int orte_iof_proxy_svc_publish(
|
||||
|
||||
|
||||
/*
|
||||
* Remove published endpoint from the server.
|
||||
* Send an "unpublish" request to the svc component
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_svc_unpublish(
|
||||
@ -93,13 +93,13 @@ int orte_iof_proxy_svc_unpublish(
|
||||
struct iovec iov;
|
||||
int rc;
|
||||
|
||||
hdr.hdr_common.hdr_type = ORTE_IOF_BASE_HDR_PUB;
|
||||
hdr.hdr_common.hdr_type = ORTE_IOF_BASE_HDR_UNPUB;
|
||||
hdr.hdr_common.hdr_status = 0;
|
||||
hdr.hdr_pub.pub_name = *name;
|
||||
hdr.hdr_pub.pub_proxy = *ORTE_PROC_MY_NAME;
|
||||
hdr.hdr_pub.pub_mask = mask;
|
||||
hdr.hdr_pub.pub_tag = tag;
|
||||
ORTE_IOF_BASE_HDR_PUB_NTOH(hdr.hdr_pub);
|
||||
ORTE_IOF_BASE_HDR_PUB_HTON(hdr.hdr_pub);
|
||||
|
||||
iov.iov_base = (IOVBASE_TYPE*)&hdr;
|
||||
iov.iov_len = sizeof(hdr);
|
||||
@ -119,17 +119,16 @@ int orte_iof_proxy_svc_unpublish(
|
||||
|
||||
|
||||
/*
|
||||
* Subscribe one or more destination process(es) to
|
||||
* one/more source process.
|
||||
* Send a "subscribe" request to the svc component
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_svc_subscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
int src_tag,
|
||||
const orte_process_name_t* dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
int dst_tag
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
int origin_tag,
|
||||
const orte_process_name_t* target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
int target_tag
|
||||
)
|
||||
{
|
||||
orte_iof_base_header_t hdr;
|
||||
@ -138,13 +137,13 @@ int orte_iof_proxy_svc_subscribe(
|
||||
|
||||
hdr.hdr_common.hdr_type = ORTE_IOF_BASE_HDR_SUB;
|
||||
hdr.hdr_common.hdr_status = 0;
|
||||
hdr.hdr_sub.src_name = *src_name;
|
||||
hdr.hdr_sub.src_mask = src_mask;
|
||||
hdr.hdr_sub.src_tag = src_tag;
|
||||
hdr.hdr_sub.dst_name = *dst_name;
|
||||
hdr.hdr_sub.dst_mask = dst_mask;
|
||||
hdr.hdr_sub.dst_tag = dst_tag;
|
||||
ORTE_IOF_BASE_HDR_SUB_NTOH(hdr.hdr_sub);
|
||||
hdr.hdr_sub.origin_name = *origin_name;
|
||||
hdr.hdr_sub.origin_mask = origin_mask;
|
||||
hdr.hdr_sub.origin_tag = origin_tag;
|
||||
hdr.hdr_sub.target_name = *target_name;
|
||||
hdr.hdr_sub.target_mask = target_mask;
|
||||
hdr.hdr_sub.target_tag = target_tag;
|
||||
ORTE_IOF_BASE_HDR_SUB_HTON(hdr.hdr_sub);
|
||||
|
||||
iov.iov_base = (IOVBASE_TYPE*)&hdr;
|
||||
iov.iov_len = sizeof(hdr);
|
||||
@ -164,16 +163,16 @@ int orte_iof_proxy_svc_subscribe(
|
||||
|
||||
|
||||
/*
|
||||
* Remove subscription message from the server.
|
||||
* Send an "unsubscribe" request to the svc component
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_svc_unsubscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
int src_tag,
|
||||
const orte_process_name_t* dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
int dst_tag
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
int origin_tag,
|
||||
const orte_process_name_t* target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
int target_tag
|
||||
)
|
||||
{
|
||||
orte_iof_base_header_t hdr;
|
||||
@ -183,13 +182,13 @@ int orte_iof_proxy_svc_unsubscribe(
|
||||
hdr.hdr_common.hdr_type = ORTE_IOF_BASE_HDR_UNSUB;
|
||||
hdr.hdr_common.hdr_reserve = (uint8_t)0;
|
||||
hdr.hdr_common.hdr_status = (int16_t)0;
|
||||
hdr.hdr_sub.src_name = *src_name;
|
||||
hdr.hdr_sub.src_mask = src_mask;
|
||||
hdr.hdr_sub.src_tag = src_tag;
|
||||
hdr.hdr_sub.dst_name = *dst_name;
|
||||
hdr.hdr_sub.dst_mask = dst_mask;
|
||||
hdr.hdr_sub.dst_tag = dst_tag;
|
||||
ORTE_IOF_BASE_HDR_SUB_NTOH(hdr.hdr_sub);
|
||||
hdr.hdr_sub.origin_name = *origin_name;
|
||||
hdr.hdr_sub.origin_mask = origin_mask;
|
||||
hdr.hdr_sub.origin_tag = origin_tag;
|
||||
hdr.hdr_sub.target_name = *target_name;
|
||||
hdr.hdr_sub.target_mask = target_mask;
|
||||
hdr.hdr_sub.target_tag = target_tag;
|
||||
ORTE_IOF_BASE_HDR_SUB_HTON(hdr.hdr_sub);
|
||||
|
||||
iov.iov_base = (IOVBASE_TYPE*)&hdr;
|
||||
iov.iov_len = sizeof(hdr);
|
||||
@ -209,13 +208,12 @@ int orte_iof_proxy_svc_unsubscribe(
|
||||
|
||||
|
||||
/*
|
||||
* Handle receipt of data/ack messages from the server
|
||||
* and forward on to the appropriate endpoint.
|
||||
* Receive messages via the RML from the svc component.
|
||||
*/
|
||||
|
||||
void orte_iof_proxy_svc_recv(
|
||||
int status,
|
||||
orte_process_name_t* src,
|
||||
orte_process_name_t* origin,
|
||||
struct iovec* msg,
|
||||
int count,
|
||||
orte_rml_tag_t tag,
|
||||
@ -223,59 +221,87 @@ void orte_iof_proxy_svc_recv(
|
||||
{
|
||||
orte_iof_base_header_t* hdr = (orte_iof_base_header_t*)msg->iov_base;
|
||||
if(NULL == msg->iov_base) {
|
||||
opal_output(0, "orte_iof_proxy_svc_recv: invalid message\n");
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_proxy_svc_recv: invalid message\n");
|
||||
return;
|
||||
}
|
||||
|
||||
/* We only receive 2 types of messages from the svc component:
|
||||
|
||||
- Messages: containing forwarded data intended to be consumed
|
||||
by endpoints in this process either representing local fd's
|
||||
or pipes to proxied processes (e.g., orted's fronting ORTE
|
||||
processes)
|
||||
|
||||
- ACKs: acknowledging data sent from this process to the svc
|
||||
component (which may have been forwarded on to other
|
||||
processes).
|
||||
*/
|
||||
|
||||
switch(hdr->hdr_common.hdr_type) {
|
||||
case ORTE_IOF_BASE_HDR_MSG:
|
||||
ORTE_IOF_BASE_HDR_MSG_NTOH(hdr->hdr_msg);
|
||||
orte_iof_proxy_svc_msg(src,&hdr->hdr_msg,(unsigned char*)(hdr+1));
|
||||
orte_iof_proxy_svc_msg(origin,&hdr->hdr_msg,(unsigned char*)(hdr+1));
|
||||
break;
|
||||
case ORTE_IOF_BASE_HDR_ACK:
|
||||
ORTE_IOF_BASE_HDR_MSG_NTOH(hdr->hdr_msg);
|
||||
orte_iof_proxy_svc_ack(src,&hdr->hdr_msg);
|
||||
orte_iof_proxy_svc_ack(origin,&hdr->hdr_msg);
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
free(hdr);
|
||||
|
||||
/* repost receive */
|
||||
|
||||
/* reset the data in the RML receive */
|
||||
mca_iof_proxy_component.proxy_iov[0].iov_base = NULL;
|
||||
mca_iof_proxy_component.proxy_iov[0].iov_len = 0;
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* Forward data message to the matching endpoint.
|
||||
* The svc component has sent data to us that matches a tag that we
|
||||
* must have previously published. Forward the data to the
|
||||
* corresponding endpoint.
|
||||
*/
|
||||
|
||||
static void orte_iof_proxy_svc_msg(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_msg_header_t* msg,
|
||||
unsigned char* data)
|
||||
{
|
||||
orte_iof_base_endpoint_t* endpoint;
|
||||
|
||||
/* Look for the endpoint corresponding to the tag in the message.
|
||||
If we don't find the endpoint, this means that we have already
|
||||
unpublished the endpoint and this message must have already
|
||||
been enroute to us when we unpublished. So just discard it. */
|
||||
endpoint = orte_iof_base_endpoint_match(ORTE_NAME_WILDCARD, ORTE_NS_CMP_NONE, msg->msg_tag);
|
||||
if(endpoint != NULL) {
|
||||
orte_iof_base_endpoint_forward(endpoint,src,msg,data);
|
||||
if (NULL != endpoint) {
|
||||
orte_iof_base_endpoint_forward(endpoint,origin,msg,data);
|
||||
/* RELEASE the endpoint because endpoint_match() RETAINed it */
|
||||
OBJ_RELEASE(endpoint);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Forward ack message to the matching endpoint.
|
||||
/*
|
||||
* The svc component has sent an ACK to us that matches a tag that we
|
||||
* must have previously published. Forward the ACK to the
|
||||
* corresponding endpoint.
|
||||
*/
|
||||
|
||||
static void orte_iof_proxy_svc_ack(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_msg_header_t* msg)
|
||||
{
|
||||
orte_iof_base_endpoint_t* endpoint;
|
||||
endpoint = orte_iof_base_endpoint_match(&msg->msg_src, ORTE_NS_CMP_ALL, msg->msg_tag);
|
||||
/* Look for the endpoint corresponding to the tag in the ACK. If
|
||||
we don't find the endpoint, this means that we have already
|
||||
unpublished the endpoint and this ACK must have already been
|
||||
enroute to us when we unpublished. So just discard it. */
|
||||
endpoint = orte_iof_base_endpoint_match(&msg->msg_origin, ORTE_NS_CMP_ALL, msg->msg_tag);
|
||||
if(endpoint != NULL) {
|
||||
orte_iof_base_endpoint_ack(endpoint,msg->msg_seq + msg->msg_len);
|
||||
/* RELEASE the endpoint because endpoint_match() RETAINed it */
|
||||
OBJ_RELEASE(endpoint);
|
||||
}
|
||||
}
|
||||
|
||||
@ -9,15 +9,14 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights resereved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
/**
|
||||
* @file
|
||||
*/
|
||||
|
||||
#ifndef MCA_IOF_PROXY_SVC_H
|
||||
#define MCA_IOF_PROXY_SVC_H
|
||||
|
||||
@ -29,8 +28,8 @@
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/**
|
||||
* Publish endpoint to forwarding service.
|
||||
/*
|
||||
* Send requests to the svc component
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_svc_publish(
|
||||
@ -44,12 +43,6 @@ int orte_iof_proxy_svc_unpublish(
|
||||
int tag
|
||||
);
|
||||
|
||||
/**
|
||||
* Subscribe one/more destination processes as
|
||||
* specified by the process name/mask to one/more
|
||||
* source processes.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_svc_subscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
@ -59,11 +52,6 @@ int orte_iof_proxy_svc_subscribe(
|
||||
int dst_tag
|
||||
);
|
||||
|
||||
/**
|
||||
* Remove subscription from forwarding
|
||||
* service.
|
||||
*/
|
||||
|
||||
int orte_iof_proxy_svc_unsubscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
@ -74,8 +62,7 @@ int orte_iof_proxy_svc_unsubscribe(
|
||||
);
|
||||
|
||||
/**
|
||||
* Dispatch messages received from forwarding
|
||||
* service.
|
||||
* Received RML messages from the svc component
|
||||
*/
|
||||
|
||||
void orte_iof_proxy_svc_recv(
|
||||
|
||||
@ -9,6 +9,8 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Sun Microsystems, Inc. All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -37,30 +39,31 @@
|
||||
orte_iof_base_module_t orte_iof_svc_module = {
|
||||
orte_iof_svc_publish,
|
||||
orte_iof_svc_unpublish,
|
||||
orte_iof_svc_push,
|
||||
orte_iof_svc_pull,
|
||||
orte_iof_svc_subscribe,
|
||||
orte_iof_svc_unsubscribe,
|
||||
orte_iof_svc_push,
|
||||
orte_iof_svc_pull,
|
||||
orte_iof_base_flush,
|
||||
orte_iof_svc_finalize,
|
||||
orte_iof_svc_ft_event
|
||||
};
|
||||
|
||||
|
||||
/**
|
||||
* Publish a local file descriptor as an endpoint that is logically
|
||||
* associated with the specified process name (e.g. master side of a
|
||||
* pipe/pty connected to a child process)
|
||||
*
|
||||
* @param name
|
||||
* @param mode
|
||||
* @param tag
|
||||
* @param fd
|
||||
/*
|
||||
* Create an endpoint for a local file descriptor and "publish" it
|
||||
* under the name of the origin process. If the publish mode is a
|
||||
* SINK, then create a publication entry for it so that incoming
|
||||
* messages can be forwarded to it.
|
||||
*
|
||||
* SOURCEs do not need to create publication records because a) the
|
||||
* endpoint will automatically wake up the event engine and read off
|
||||
* the fd whenever there is data available, and b) this data is then
|
||||
* automatically sent to the iof svc component for possible
|
||||
* forwarding.
|
||||
*/
|
||||
|
||||
int orte_iof_svc_publish(
|
||||
const orte_process_name_t* name,
|
||||
const orte_process_name_t* origin,
|
||||
orte_iof_base_mode_t mode,
|
||||
orte_iof_base_tag_t tag,
|
||||
int fd)
|
||||
@ -69,18 +72,18 @@ int orte_iof_svc_publish(
|
||||
|
||||
/* setup a local endpoint to reflect registration */
|
||||
rc = orte_iof_base_endpoint_create(
|
||||
name,
|
||||
origin,
|
||||
mode,
|
||||
tag,
|
||||
fd);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* publish endpoint */
|
||||
if(mode == ORTE_IOF_SINK) {
|
||||
if (ORTE_IOF_SINK == mode) {
|
||||
rc = orte_iof_svc_pub_create(
|
||||
name,
|
||||
origin,
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
tag);
|
||||
@ -89,181 +92,174 @@ int orte_iof_svc_publish(
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Remove all registrations matching the specified process
|
||||
* name, mask and tag values.
|
||||
*
|
||||
* @param name
|
||||
* @param mask
|
||||
* @param tag
|
||||
*
|
||||
/*
|
||||
* Remove all registrations matching the specified origin process
|
||||
* name, mask and tag values (where, here in the svc component, origin
|
||||
* should usually be just this process -- ths svc component is
|
||||
* unlikely to act as an IOF proxy for any other processes like the
|
||||
* orted does).
|
||||
*/
|
||||
|
||||
int orte_iof_svc_unpublish(
|
||||
const orte_process_name_t* name,
|
||||
const orte_process_name_t* origin,
|
||||
orte_ns_cmp_bitmask_t mask,
|
||||
orte_iof_base_tag_t tag)
|
||||
{
|
||||
int rc;
|
||||
rc = orte_iof_svc_pub_delete(
|
||||
name,
|
||||
origin,
|
||||
ORTE_PROC_MY_NAME,
|
||||
mask,
|
||||
tag);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* setup a local endpoint to reflect registration */
|
||||
rc = orte_iof_base_endpoint_delete(
|
||||
name,
|
||||
origin,
|
||||
mask,
|
||||
tag);
|
||||
|
||||
|
||||
return rc;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Explicitly push data from the specified file descriptor
|
||||
* to the indicated set of peers.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor.
|
||||
* to the indicated set of SINK peers.
|
||||
*/
|
||||
|
||||
int orte_iof_svc_push(
|
||||
const orte_process_name_t* dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
orte_iof_base_tag_t dst_tag,
|
||||
const orte_process_name_t* sink_name,
|
||||
orte_ns_cmp_bitmask_t sink_mask,
|
||||
orte_iof_base_tag_t sink_tag,
|
||||
int fd)
|
||||
{
|
||||
int rc;
|
||||
|
||||
/* setup a subscription */
|
||||
/* Setup a subscription. This will be matched against a publish
|
||||
of a SINK from a remote process. */
|
||||
rc = orte_iof_svc_sub_create(
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
dst_tag,
|
||||
dst_name,
|
||||
dst_mask,
|
||||
dst_tag);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
sink_tag,
|
||||
sink_name,
|
||||
sink_mask,
|
||||
sink_tag);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* setup a local endpoint to reflect registration */
|
||||
/* Setup a local endpoint to reflect registration. This will
|
||||
enter the fd into the event engine and wakeup when there is
|
||||
data to read. The data will be put in an IOF fragment and RML
|
||||
send to iof_svc_proxy_recv() (i.e., in this module!) for
|
||||
handling (i.e., matching and forwarding to the publish(es) that
|
||||
was(were) matched to the above subscription).
|
||||
|
||||
Create this endpoint *after* we make the above subscription so
|
||||
that it is not found and attached to the subscription.
|
||||
Instead, data that is consumed by the event engine callbacks
|
||||
will be RML-sent to iof_svc_proxy_recv(), as described
|
||||
above. */
|
||||
rc = orte_iof_base_endpoint_create(
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_IOF_SOURCE,
|
||||
dst_tag,
|
||||
sink_tag,
|
||||
fd);
|
||||
return rc;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Explicitly pull data from the specified set of peers
|
||||
/*
|
||||
* Explicitly pull data from the specified set of SOURCE peers
|
||||
* and dump to the indicated file descriptor.
|
||||
*
|
||||
* @param dst_name Name used to qualify set of peers.
|
||||
* @param dst_mask Mask that specified how name is interpreted.
|
||||
* @param dst_tag Match a specific peer endpoint.
|
||||
* @param fd Local file descriptor.
|
||||
*/
|
||||
|
||||
int orte_iof_svc_pull(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t* source_name,
|
||||
orte_ns_cmp_bitmask_t source_mask,
|
||||
orte_iof_base_tag_t source_tag,
|
||||
int fd)
|
||||
{
|
||||
int rc;
|
||||
|
||||
/* setup a local endpoint */
|
||||
/* setup a local endpoint -- *before* we create the subscription
|
||||
so that the subscription will find the endpoint and attach it
|
||||
to the subscription */
|
||||
rc = orte_iof_base_endpoint_create(
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_IOF_SINK,
|
||||
src_tag,
|
||||
source_tag,
|
||||
fd);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* create a subscription */
|
||||
rc = orte_iof_svc_sub_create(
|
||||
src_name,
|
||||
src_mask,
|
||||
src_tag,
|
||||
source_name,
|
||||
source_mask,
|
||||
source_tag,
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
src_tag);
|
||||
source_tag);
|
||||
return rc;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Setup buffering for a specified set of endpoints.
|
||||
*/
|
||||
|
||||
int orte_iof_svc_buffer(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
size_t buffer_size)
|
||||
{
|
||||
/* send a message to the server indicating this set of connections should be buffered */
|
||||
return ORTE_ERROR;
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* Subscribe to receive a callback on receipt of data
|
||||
* from a specified set of peers.
|
||||
* from a specified set of origin peers.
|
||||
*/
|
||||
|
||||
int orte_iof_svc_subscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag,
|
||||
orte_iof_base_callback_fn_t cbfunc,
|
||||
void* cbdata)
|
||||
{
|
||||
int rc;
|
||||
|
||||
/* create a local registration to reflect the callback */
|
||||
rc = orte_iof_base_callback_create(ORTE_PROC_MY_NAME,src_tag,cbfunc,cbdata);
|
||||
if(rc != ORTE_SUCCESS)
|
||||
rc = orte_iof_base_callback_create(ORTE_PROC_MY_NAME, origin_tag,
|
||||
cbfunc, cbdata);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* setup local subscription */
|
||||
rc = orte_iof_svc_sub_create(
|
||||
src_name,
|
||||
src_mask,
|
||||
src_tag,
|
||||
origin_name,
|
||||
origin_mask,
|
||||
origin_tag,
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
src_tag);
|
||||
origin_tag);
|
||||
return rc;
|
||||
}
|
||||
|
||||
int orte_iof_svc_unsubscribe(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag)
|
||||
const orte_process_name_t* origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag)
|
||||
{
|
||||
int rc;
|
||||
|
||||
/* delete local subscription */
|
||||
rc = orte_iof_svc_sub_delete(
|
||||
src_name,
|
||||
src_mask,
|
||||
src_tag,
|
||||
origin_name,
|
||||
origin_mask,
|
||||
origin_tag,
|
||||
ORTE_PROC_MY_NAME,
|
||||
ORTE_NS_CMP_ALL,
|
||||
src_tag);
|
||||
if(ORTE_SUCCESS != rc)
|
||||
origin_tag);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* cleanup any locally registered callback */
|
||||
return orte_iof_base_callback_delete(ORTE_PROC_MY_NAME,src_tag);
|
||||
return orte_iof_base_callback_delete(ORTE_PROC_MY_NAME, origin_tag);
|
||||
}
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -17,6 +18,30 @@
|
||||
*/
|
||||
/**
|
||||
* @file
|
||||
*
|
||||
* The svc IOF component is used in HNP processes only. It is the
|
||||
* "hub" for all IOF activity, meaning that *all* IOF traffic is
|
||||
* routed to the svc component, and this component figures out where
|
||||
* it is supposed to go from there. Specifically: there is *no*
|
||||
* direct proxy-to-proxy IOF communication. If a proxy/orted wants to
|
||||
* get a stream from another proxy/orted, the stream will go
|
||||
* proxy/orted -> svc/HNP -> proxy/orted.
|
||||
*
|
||||
* The svc IOF component does two things: 1. forward fragments between
|
||||
* file descriptors and streams, and 2. maintain forwarding tables to
|
||||
* "route" incomding fragments to outgoing destinations (both file
|
||||
* descriptors and other published streams).
|
||||
*
|
||||
* The svc IOF component maintains tables of all publications and all
|
||||
* subscriptions. Subscriptions can have a list of publications
|
||||
* and/or endpoints to forward incoming fragments to.
|
||||
*
|
||||
*
|
||||
*
|
||||
* Important: this component is designed to work with the proxy IOF
|
||||
* component only. If we ever do a different IOF implementation
|
||||
* scheme, it is likely that only some of this component will be
|
||||
* useful for cannibalisation (if any at all).
|
||||
*/
|
||||
#ifndef ORTE_IOF_SVC_H
|
||||
#define ORTE_IOF_SVC_H
|
||||
@ -103,17 +128,6 @@ int orte_iof_svc_pull(
|
||||
int fd
|
||||
);
|
||||
|
||||
/**
|
||||
* Setup buffering for a specified set of endpoints.
|
||||
*/
|
||||
|
||||
int orte_iof_svc_buffer(
|
||||
const orte_process_name_t* src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
size_t buffer_size
|
||||
);
|
||||
|
||||
/*
|
||||
* Subscribe to receive a callback on receipt of data
|
||||
* from a specified set of peers.
|
||||
@ -142,7 +156,6 @@ int orte_iof_svc_finalize(void);
|
||||
*/
|
||||
struct orte_iof_svc_component_t {
|
||||
orte_iof_base_component_t super;
|
||||
int svc_debug;
|
||||
opal_list_t svc_published;
|
||||
opal_list_t svc_subscribed;
|
||||
opal_mutex_t svc_lock;
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Sun Microsystems, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -74,23 +75,12 @@ orte_iof_svc_component_t mca_iof_svc_component = {
|
||||
}
|
||||
};
|
||||
|
||||
static int orte_iof_svc_param_register_int(
|
||||
const char* param_name,
|
||||
int default_value)
|
||||
{
|
||||
int id = mca_base_param_register_int("iof","svc",param_name,NULL,default_value);
|
||||
int param_value = default_value;
|
||||
mca_base_param_lookup_int(id,¶m_value);
|
||||
return param_value;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* component open/close/init function
|
||||
*/
|
||||
static int orte_iof_svc_open(void)
|
||||
{
|
||||
mca_iof_svc_component.svc_debug = orte_iof_svc_param_register_int("debug", 1);
|
||||
/* Nothing to do */
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
@ -122,8 +112,31 @@ static int orte_iof_svc_close(void)
|
||||
static void
|
||||
orte_iof_svc_exception_handler(const orte_process_name_t* peer, orte_rml_exception_t reason)
|
||||
{
|
||||
orte_iof_base_endpoint_t *endpoint;
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"iof svc exception handler! [%lu,%lu,%lu]\n",
|
||||
ORTE_NAME_ARGS(peer));
|
||||
|
||||
/* If we detect an exception on the RML connection to a peer,
|
||||
delete all of its subscriptions and publications. Note that
|
||||
exceptions can be detected during a normal RML shutdown; they
|
||||
are recoverable events (no need to abort). */
|
||||
orte_iof_svc_sub_delete_all(peer);
|
||||
orte_iof_svc_pub_delete_all(peer);
|
||||
opal_output(orte_iof_base.iof_output, "deleted all pubs and subs\n");
|
||||
|
||||
/* Find any streams on any endpoints for this peer and close them */
|
||||
while (NULL !=
|
||||
(endpoint = orte_iof_base_endpoint_match(peer, ORTE_NS_CMP_ALL,
|
||||
ORTE_IOF_ANY))) {
|
||||
orte_iof_base_endpoint_closed(endpoint);
|
||||
};
|
||||
/* Then delete them */
|
||||
while (ORTE_SUCCESS ==
|
||||
orte_iof_base_endpoint_delete(peer, ORTE_NS_CMP_ALL, ORTE_IOF_ANY)) {
|
||||
continue;
|
||||
}
|
||||
opal_output(orte_iof_base.iof_output, "done with exception handler\n");
|
||||
}
|
||||
|
||||
|
||||
@ -162,7 +175,8 @@ orte_iof_svc_init(int* priority, bool *allow_multi_user_threads, bool *have_hidd
|
||||
NULL
|
||||
);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
opal_output(0, "orte_iof_svc_init: unable to post non-blocking recv");
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_init: unable to post non-blocking recv");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
|
||||
@ -9,6 +9,7 @@
|
||||
* University of Stuttgart. All rights reserved.
|
||||
* Copyright (c) 2004-2005 The Regents of the University of California.
|
||||
* All rights reserved.
|
||||
* Copyright (c) 2007 Cisco, Inc. All rights reserved.
|
||||
* $COPYRIGHT$
|
||||
*
|
||||
* Additional copyrights may follow
|
||||
@ -31,7 +32,7 @@
|
||||
|
||||
|
||||
static void orte_iof_svc_proxy_msg(const orte_process_name_t*, orte_iof_base_msg_header_t*, unsigned char*);
|
||||
static void orte_iof_svc_proxy_ack(const orte_process_name_t*, orte_iof_base_msg_header_t*);
|
||||
static void orte_iof_svc_proxy_ack(const orte_process_name_t*, orte_iof_base_msg_header_t*, bool do_close);
|
||||
static void orte_iof_svc_proxy_pub(const orte_process_name_t*, orte_iof_base_pub_header_t*);
|
||||
static void orte_iof_svc_proxy_unpub(const orte_process_name_t*, orte_iof_base_pub_header_t*);
|
||||
static void orte_iof_svc_proxy_sub(const orte_process_name_t*, orte_iof_base_sub_header_t*);
|
||||
@ -40,7 +41,7 @@ static void orte_iof_svc_proxy_unsub(const orte_process_name_t*, orte_iof_base_s
|
||||
|
||||
|
||||
/**
|
||||
* Callback function from OOB on receipt of IOF request.
|
||||
* Callback function from RML on receipt of IOF request.
|
||||
*
|
||||
* @param status (IN) Completion status.
|
||||
* @param peer (IN) Opaque name of peer process.
|
||||
@ -48,8 +49,8 @@ static void orte_iof_svc_proxy_unsub(const orte_process_name_t*, orte_iof_base_s
|
||||
* @param count (IN) Number of elements in iovec array.
|
||||
* @param tag (IN) User defined tag for matching send/recv.
|
||||
* @param cbdata (IN) User data.
|
||||
*/
|
||||
|
||||
*/
|
||||
|
||||
void orte_iof_svc_proxy_recv(
|
||||
int status,
|
||||
orte_process_name_t* peer,
|
||||
@ -67,31 +68,51 @@ void orte_iof_svc_proxy_recv(
|
||||
switch(hdr->hdr_common.hdr_type) {
|
||||
case ORTE_IOF_BASE_HDR_MSG:
|
||||
ORTE_IOF_BASE_HDR_MSG_NTOH(hdr->hdr_msg);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_recv: HDR_MSG\n");
|
||||
orte_iof_svc_proxy_msg(peer, &hdr->hdr_msg,
|
||||
((unsigned char*)iov[0].iov_base)+sizeof(orte_iof_base_header_t));
|
||||
break;
|
||||
case ORTE_IOF_BASE_HDR_ACK:
|
||||
ORTE_IOF_BASE_HDR_MSG_NTOH(hdr->hdr_msg);
|
||||
orte_iof_svc_proxy_ack(peer, &hdr->hdr_msg);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_recv: HDR_ACK\n");
|
||||
orte_iof_svc_proxy_ack(peer, &hdr->hdr_msg, false);
|
||||
break;
|
||||
case ORTE_IOF_BASE_HDR_CLOSE:
|
||||
ORTE_IOF_BASE_HDR_MSG_NTOH(hdr->hdr_msg);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_recv: HDR_CLOSE\n");
|
||||
orte_iof_svc_proxy_ack(peer, &hdr->hdr_msg, true);
|
||||
break;
|
||||
case ORTE_IOF_BASE_HDR_PUB:
|
||||
ORTE_IOF_BASE_HDR_PUB_NTOH(hdr->hdr_pub);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_recv: HDR_PUB\n");
|
||||
orte_iof_svc_proxy_pub(peer, &hdr->hdr_pub);
|
||||
break;
|
||||
case ORTE_IOF_BASE_HDR_UNPUB:
|
||||
ORTE_IOF_BASE_HDR_PUB_NTOH(hdr->hdr_pub);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_recv: HDR_UNPUB\n");
|
||||
orte_iof_svc_proxy_unpub(peer, &hdr->hdr_pub);
|
||||
break;
|
||||
case ORTE_IOF_BASE_HDR_SUB:
|
||||
ORTE_IOF_BASE_HDR_SUB_NTOH(hdr->hdr_sub);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_recv: HDR_SUB\n");
|
||||
orte_iof_svc_proxy_sub(peer, &hdr->hdr_sub);
|
||||
break;
|
||||
case ORTE_IOF_BASE_HDR_UNSUB:
|
||||
ORTE_IOF_BASE_HDR_SUB_NTOH(hdr->hdr_sub);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_recv: HDR_UNSUB\n");
|
||||
orte_iof_svc_proxy_unsub(peer, &hdr->hdr_sub);
|
||||
break;
|
||||
default:
|
||||
opal_output(0, "orte_iof_svc_recv: invalid message type: %d\n", hdr->hdr_common.hdr_type);
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_recv: invalid message type: %d (ignored)\n",
|
||||
hdr->hdr_common.hdr_type);
|
||||
break;
|
||||
}
|
||||
|
||||
@ -106,7 +127,7 @@ done:
|
||||
/**
|
||||
* Release resources when ack completed.
|
||||
*/
|
||||
static void orte_iof_svc_ack_send_cb(
|
||||
static void ack_send_cb(
|
||||
int status,
|
||||
orte_process_name_t* peer,
|
||||
struct iovec* msg,
|
||||
@ -120,7 +141,7 @@ static void orte_iof_svc_ack_send_cb(
|
||||
ORTE_ERROR_LOG(status);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Receive a data message. Check the subscription list for a match
|
||||
* on the source - and on matches forward to any published endpoints
|
||||
@ -128,15 +149,15 @@ static void orte_iof_svc_ack_send_cb(
|
||||
*/
|
||||
|
||||
static void orte_iof_svc_proxy_msg(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_msg_header_t* hdr,
|
||||
unsigned char* data)
|
||||
{
|
||||
opal_list_item_t* item;
|
||||
bool forward = false;
|
||||
if(mca_iof_svc_component.svc_debug > 1) {
|
||||
opal_output(0, "orte_iof_svc_proxy_msg: tag %d seq %d\n",hdr->msg_tag,hdr->msg_seq);
|
||||
}
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"orte_iof_svc_proxy_msg: tag %d seq %d",
|
||||
hdr->msg_tag,hdr->msg_seq);
|
||||
|
||||
/* dispatch based on subscription list */
|
||||
OPAL_THREAD_LOCK(&mca_iof_svc_component.svc_lock);
|
||||
@ -145,45 +166,50 @@ static void orte_iof_svc_proxy_msg(
|
||||
item = opal_list_get_next(item)) {
|
||||
orte_iof_svc_sub_t* sub = (orte_iof_svc_sub_t*)item;
|
||||
|
||||
/* tags match */
|
||||
if(sub->src_tag != hdr->msg_tag && hdr->msg_tag != ORTE_IOF_ANY)
|
||||
/* if the tags don't match, skip this subscription */
|
||||
if(sub->origin_tag != hdr->msg_tag && hdr->msg_tag != ORTE_IOF_ANY) {
|
||||
continue;
|
||||
}
|
||||
|
||||
/* source match */
|
||||
if(orte_ns.compare_fields(sub->src_mask,&sub->src_name,&hdr->msg_src) == 0) {
|
||||
if(mca_iof_svc_component.svc_debug > 1) {
|
||||
opal_output(0, "[%lu,%lu,%lu] orte_iof_svc_proxy_msg: tag %d sequence %d\n",
|
||||
ORTE_NAME_ARGS(&sub->src_name),hdr->msg_tag,hdr->msg_seq);
|
||||
}
|
||||
orte_iof_svc_sub_forward(sub,src,hdr,data,&forward);
|
||||
/* if the subscription origin doesn't match the message's
|
||||
origin, skip this subscription */
|
||||
if(orte_ns.compare_fields(sub->origin_mask,&sub->origin_name,&hdr->msg_origin) == 0) {
|
||||
opal_output(orte_iof_base.iof_output, "sub origin [%lu,%lu,%lu], msg origin [%lu,%lu,%lu], msg proxy [%lu,%lu,%lu] orte_iof_svc_proxy_msg: tag %d sequence %d, len %d\n",
|
||||
ORTE_NAME_ARGS(&sub->origin_name),
|
||||
ORTE_NAME_ARGS(&hdr->msg_origin),
|
||||
ORTE_NAME_ARGS(&hdr->msg_proxy),
|
||||
hdr->msg_tag, hdr->msg_seq, hdr->msg_len);
|
||||
/* Everthing matched -- forward the message */
|
||||
orte_iof_svc_sub_forward(sub,peer,hdr,data,&forward);
|
||||
}
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&mca_iof_svc_component.svc_lock);
|
||||
|
||||
/* if there is no one to forward to - go ahead and ack */
|
||||
/* If there was no one to forward to, then we effectively drop it.
|
||||
But ACK it so that the sender doesn't block. */
|
||||
if(forward == false) {
|
||||
orte_iof_base_frag_t* frag;
|
||||
int rc;
|
||||
|
||||
|
||||
ORTE_IOF_BASE_FRAG_ALLOC(frag,rc);
|
||||
if(NULL == frag) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
frag->frag_hdr.hdr_msg = *hdr;
|
||||
frag->frag_hdr.hdr_common.hdr_type = ORTE_IOF_BASE_HDR_ACK;
|
||||
frag->frag_iov[0].iov_base = (IOVBASE_TYPE*)&frag->frag_hdr;
|
||||
frag->frag_iov[0].iov_len = sizeof(frag->frag_hdr);
|
||||
ORTE_IOF_BASE_HDR_MSG_HTON(frag->frag_hdr.hdr_msg);
|
||||
|
||||
|
||||
rc = orte_rml.send_nb(
|
||||
&hdr->msg_proxy,
|
||||
frag->frag_iov,
|
||||
1,
|
||||
ORTE_RML_TAG_IOF_SVC,
|
||||
0,
|
||||
orte_iof_svc_ack_send_cb,
|
||||
ack_send_cb,
|
||||
frag);
|
||||
if(rc < 0) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
@ -191,189 +217,87 @@ static void orte_iof_svc_proxy_msg(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Received an acknowledgment from an endpoint - forward on
|
||||
* towards the source if all other endpoints have also
|
||||
* acknowledged the data.
|
||||
*/
|
||||
|
||||
static void orte_iof_svc_proxy_ack(
|
||||
const orte_process_name_t* src,
|
||||
orte_iof_base_msg_header_t* hdr)
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_msg_header_t* hdr,
|
||||
bool do_close)
|
||||
{
|
||||
opal_list_item_t *s_item;
|
||||
uint32_t seq_min = hdr->msg_seq + hdr->msg_len;
|
||||
union {
|
||||
uint32_t uval;
|
||||
void *vval;
|
||||
} value;
|
||||
|
||||
if(mca_iof_svc_component.svc_debug > 1) {
|
||||
opal_output(0, "orte_iof_svc_proxy_ack");
|
||||
}
|
||||
|
||||
/* for each of the subscriptions that match the source of the data:
|
||||
* (1) find all forwarding entries that match the source of the ack
|
||||
* (2) update their sequence number
|
||||
* (3) find the minimum sequence number across all endpoints
|
||||
*/
|
||||
|
||||
OPAL_THREAD_LOCK(&mca_iof_svc_component.svc_lock);
|
||||
for(s_item = opal_list_get_first(&mca_iof_svc_component.svc_subscribed);
|
||||
s_item != opal_list_get_end(&mca_iof_svc_component.svc_subscribed);
|
||||
s_item = opal_list_get_next(s_item)) {
|
||||
|
||||
orte_iof_svc_sub_t* sub = (orte_iof_svc_sub_t*)s_item;
|
||||
opal_list_item_t *f_item;
|
||||
|
||||
if (orte_ns.compare_fields(sub->src_mask,&sub->src_name,&hdr->msg_src) != 0 ||
|
||||
sub->src_tag != hdr->msg_tag) {
|
||||
continue;
|
||||
}
|
||||
|
||||
/* look for this endpoint in the forwarding table */
|
||||
for(f_item = opal_list_get_first(&sub->sub_forward);
|
||||
f_item != opal_list_get_end(&sub->sub_forward);
|
||||
f_item = opal_list_get_next(f_item)) {
|
||||
orte_iof_svc_fwd_t* fwd = (orte_iof_svc_fwd_t*)f_item;
|
||||
orte_iof_svc_pub_t* pub = fwd->fwd_pub;
|
||||
if (orte_ns.compare_fields(pub->pub_mask,&pub->pub_name,src) == 0 ||
|
||||
orte_ns.compare_fields(ORTE_NS_CMP_ALL,&pub->pub_proxy,src) == 0) {
|
||||
value.uval = hdr->msg_seq + hdr->msg_len;
|
||||
orte_hash_table_set_proc(&fwd->fwd_seq,
|
||||
&hdr->msg_src, &value.vval);
|
||||
} else {
|
||||
value.vval = orte_hash_table_get_proc(&fwd->fwd_seq,
|
||||
&hdr->msg_src);
|
||||
if(value.uval < seq_min) {
|
||||
seq_min = value.uval;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&mca_iof_svc_component.svc_lock);
|
||||
|
||||
/* if all destination endpoints have acknowledged up to this
|
||||
* sequence number ack the source
|
||||
*/
|
||||
if(seq_min == hdr->msg_seq+hdr->msg_len) {
|
||||
|
||||
if(orte_ns.compare_fields(ORTE_NS_CMP_ALL,orte_process_info.my_name,&hdr->msg_src) == 0) {
|
||||
orte_iof_base_endpoint_t* endpoint;
|
||||
/*
|
||||
* Local delivery
|
||||
*/
|
||||
endpoint = orte_iof_base_endpoint_match(&hdr->msg_src, ORTE_NS_CMP_ALL, hdr->msg_tag);
|
||||
if(endpoint != NULL) {
|
||||
orte_iof_base_endpoint_ack(endpoint, hdr->msg_seq + hdr->msg_len);
|
||||
OBJ_RELEASE(endpoint);
|
||||
}
|
||||
|
||||
} else {
|
||||
|
||||
/*
|
||||
* forward on to source
|
||||
*/
|
||||
orte_iof_base_frag_t* frag;
|
||||
int rc;
|
||||
|
||||
ORTE_IOF_BASE_FRAG_ALLOC(frag,rc);
|
||||
if(NULL == frag) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
return;
|
||||
}
|
||||
|
||||
frag->frag_hdr.hdr_msg = *hdr;
|
||||
frag->frag_iov[0].iov_base = (IOVBASE_TYPE*)&frag->frag_hdr;
|
||||
frag->frag_iov[0].iov_len = sizeof(frag->frag_hdr);
|
||||
ORTE_IOF_BASE_HDR_MSG_HTON(frag->frag_hdr.hdr_msg);
|
||||
|
||||
rc = orte_rml.send_nb(
|
||||
&hdr->msg_proxy,
|
||||
frag->frag_iov,
|
||||
1,
|
||||
ORTE_RML_TAG_IOF_SVC,
|
||||
0,
|
||||
orte_iof_svc_ack_send_cb,
|
||||
frag);
|
||||
if(rc < 0) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
}
|
||||
}
|
||||
}
|
||||
orte_iof_svc_sub_ack(peer, hdr, do_close);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create an entry to represent the published endpoint. This
|
||||
* also checks to see if the endpoint matches any pending
|
||||
* subscriptions.
|
||||
* A remote process has announced that it has an endpoint that it is
|
||||
* making available. Create an entry for it, and also check to see if
|
||||
* the new publication matches any pending subscriptions.
|
||||
*/
|
||||
|
||||
static void orte_iof_svc_proxy_pub(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_pub_header_t* hdr)
|
||||
{
|
||||
int rc;
|
||||
if(mca_iof_svc_component.svc_debug > 1) {
|
||||
opal_output(0, "orte_iof_svc_proxy_pub");
|
||||
}
|
||||
opal_output(orte_iof_base.iof_output, "orte_iof_svc_proxy_pub: mask %d, tag %d, proc [%lu,%lu,%lu], proxy [%lu,%lu,%lu]",
|
||||
hdr->pub_mask, hdr->pub_tag,
|
||||
ORTE_NAME_ARGS(&hdr->pub_name),
|
||||
ORTE_NAME_ARGS(&hdr->pub_proxy));
|
||||
|
||||
rc = orte_iof_svc_pub_create(
|
||||
&hdr->pub_name,
|
||||
&hdr->pub_proxy,
|
||||
hdr->pub_mask,
|
||||
hdr->pub_tag);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
*
|
||||
/**
|
||||
* Opposite of publish -- remove any previous publications and unmap
|
||||
* it from any subscriptions that were using it.
|
||||
*/
|
||||
|
||||
static void orte_iof_svc_proxy_unpub(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_pub_header_t* hdr)
|
||||
{
|
||||
int rc;
|
||||
if(mca_iof_svc_component.svc_debug > 1) {
|
||||
opal_output(0, "orte_iof_svc_proxy_unpub");
|
||||
}
|
||||
opal_output(orte_iof_base.iof_output, "orte_iof_svc_proxy_unpub: mask %d, tag %d, proc [%lu,%lu,%lu], proxy [%lu,%lu,%lu]",
|
||||
hdr->pub_mask, hdr->pub_tag,
|
||||
ORTE_NAME_ARGS(&hdr->pub_name),
|
||||
ORTE_NAME_ARGS(&hdr->pub_proxy));
|
||||
|
||||
rc = orte_iof_svc_pub_delete(
|
||||
&hdr->pub_name,
|
||||
&hdr->pub_proxy,
|
||||
hdr->pub_mask,
|
||||
hdr->pub_tag);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
if (ORTE_SUCCESS != rc && ORTE_ERR_NOT_FOUND != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a subscription entry. A subscription entry
|
||||
* determines the set of source(s) that will forward
|
||||
* to any matching published endpoints.
|
||||
* Create a subscription entry. A subscription entry determines the
|
||||
* set of origin(s) that will forward to any matching published
|
||||
* endpoint targets.
|
||||
*/
|
||||
|
||||
static void orte_iof_svc_proxy_sub(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_sub_header_t* hdr)
|
||||
{
|
||||
int rc;
|
||||
if(mca_iof_svc_component.svc_debug > 1) {
|
||||
opal_output(0, "orte_iof_svc_proxy_sub");
|
||||
}
|
||||
opal_output(orte_iof_base.iof_output, "orte_iof_svc_proxy_sub");
|
||||
|
||||
rc = orte_iof_svc_sub_create(
|
||||
&hdr->src_name,
|
||||
hdr->src_mask,
|
||||
hdr->src_tag,
|
||||
&hdr->dst_name,
|
||||
hdr->dst_mask,
|
||||
hdr->dst_tag);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
&hdr->origin_name,
|
||||
hdr->origin_mask,
|
||||
hdr->origin_tag,
|
||||
&hdr->target_name,
|
||||
hdr->target_mask,
|
||||
hdr->target_tag);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
}
|
||||
}
|
||||
@ -383,22 +307,20 @@ static void orte_iof_svc_proxy_sub(
|
||||
*/
|
||||
|
||||
static void orte_iof_svc_proxy_unsub(
|
||||
const orte_process_name_t* src,
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_sub_header_t* hdr)
|
||||
{
|
||||
int rc;
|
||||
if(mca_iof_svc_component.svc_debug > 1) {
|
||||
opal_output(0, "orte_iof_svc_proxy_unsub");
|
||||
}
|
||||
opal_output(orte_iof_base.iof_output, "orte_iof_svc_proxy_unsub");
|
||||
|
||||
rc = orte_iof_svc_sub_delete(
|
||||
&hdr->src_name,
|
||||
hdr->src_mask,
|
||||
hdr->src_tag,
|
||||
&hdr->dst_name,
|
||||
hdr->dst_mask,
|
||||
hdr->dst_tag);
|
||||
if(rc != ORTE_SUCCESS) {
|
||||
&hdr->origin_name,
|
||||
hdr->origin_mask,
|
||||
hdr->origin_tag,
|
||||
&hdr->target_name,
|
||||
hdr->target_mask,
|
||||
hdr->target_tag);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
}
|
||||
}
|
||||
|
||||
@ -15,9 +15,7 @@
|
||||
*
|
||||
* $HEADER$
|
||||
*/
|
||||
/**
|
||||
* @file
|
||||
*/
|
||||
|
||||
#ifndef ORTE_IOF_SVC_PROXY_H
|
||||
#define ORTE_IOF_SVC_PROXY_H
|
||||
|
||||
@ -32,7 +30,7 @@ extern "C" {
|
||||
|
||||
|
||||
/**
|
||||
* Callback function from OOB on receipt of IOF request.
|
||||
* Callback function from RML on receipt of IOF request.
|
||||
*
|
||||
* @param status (IN) Completion status.
|
||||
* @param peer (IN) Opaque name of peer process.
|
||||
@ -40,8 +38,8 @@ extern "C" {
|
||||
* @param count (IN) Number of elements in iovec array.
|
||||
* @param tag (IN) User defined tag for matching send/recv.
|
||||
* @param cbdata (IN) User data.
|
||||
*/
|
||||
|
||||
*/
|
||||
|
||||
void orte_iof_svc_proxy_recv(
|
||||
int status,
|
||||
orte_process_name_t* peer,
|
||||
|
||||
@ -9,21 +9,10 @@
|
||||
|
||||
|
||||
|
||||
static void orte_iof_svc_pub_construct(orte_iof_svc_pub_t* publish)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
static void orte_iof_svc_pub_destruct(orte_iof_svc_pub_t* publish)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
OBJ_CLASS_INSTANCE(
|
||||
orte_iof_svc_pub_t,
|
||||
opal_list_item_t,
|
||||
orte_iof_svc_pub_construct,
|
||||
orte_iof_svc_pub_destruct);
|
||||
NULL, NULL);
|
||||
|
||||
|
||||
/**
|
||||
@ -31,7 +20,7 @@ OBJ_CLASS_INSTANCE(
|
||||
* (2) Lookup any subscriptions that match and install on the
|
||||
* subscription as a destination endpoint.
|
||||
*/
|
||||
|
||||
|
||||
int orte_iof_svc_pub_create(
|
||||
const orte_process_name_t *pub_name,
|
||||
const orte_process_name_t *pub_proxy,
|
||||
@ -43,7 +32,7 @@ int orte_iof_svc_pub_create(
|
||||
|
||||
OPAL_THREAD_LOCK(&mca_iof_svc_component.svc_lock);
|
||||
|
||||
/* has this endpoint already been published */
|
||||
/* has matching publish already been created? */
|
||||
for(item = opal_list_get_first(&mca_iof_svc_component.svc_published);
|
||||
item != opal_list_get_end(&mca_iof_svc_component.svc_published);
|
||||
item = opal_list_get_next(item)) {
|
||||
@ -56,13 +45,18 @@ int orte_iof_svc_pub_create(
|
||||
}
|
||||
}
|
||||
|
||||
/* create a new entry for this endponit */
|
||||
/* create a new publish entry; associate it with the corresponding
|
||||
endpoint */
|
||||
pub = OBJ_NEW(orte_iof_svc_pub_t);
|
||||
pub->pub_name = *pub_name;
|
||||
pub->pub_proxy = *pub_proxy;
|
||||
pub->pub_mask = pub_mask;
|
||||
pub->pub_tag = pub_tag;
|
||||
pub->pub_endpoint = orte_iof_base_endpoint_match(pub_name,pub_mask,pub_tag);
|
||||
pub->pub_endpoint =
|
||||
orte_iof_base_endpoint_match(pub_name,pub_mask,pub_tag);
|
||||
opal_output(orte_iof_base.iof_output, "created svc pub, name [%lu,%lu,%lu], proxy [%lu,%lu,%lu], tag %d / mask %x, endpoint %p\n",
|
||||
ORTE_NAME_ARGS(pub_name), ORTE_NAME_ARGS(pub_proxy),
|
||||
pub_tag, pub_mask, (char*) pub->pub_endpoint);
|
||||
|
||||
/* append this published endpoint to any matching subscription */
|
||||
for(item = opal_list_get_first(&mca_iof_svc_component.svc_subscribed);
|
||||
@ -80,9 +74,9 @@ int orte_iof_svc_pub_create(
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/**
|
||||
* Look for a matching endpoint.
|
||||
* Look for a matching publish
|
||||
*/
|
||||
|
||||
orte_iof_svc_pub_t* orte_iof_svc_pub_lookup(
|
||||
@ -110,7 +104,7 @@ orte_iof_svc_pub_t* orte_iof_svc_pub_lookup(
|
||||
* Remove the published endpoint and cleanup any associated
|
||||
* forwarding entries.
|
||||
*/
|
||||
|
||||
|
||||
int orte_iof_svc_pub_delete(
|
||||
const orte_process_name_t *pub_name,
|
||||
const orte_process_name_t *pub_proxy,
|
||||
@ -140,7 +134,7 @@ int orte_iof_svc_pub_delete(
|
||||
OPAL_THREAD_UNLOCK(&mca_iof_svc_component.svc_lock);
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*
|
||||
* Remove all publications associated w/ the given process name.
|
||||
|
||||
@ -21,10 +21,11 @@
|
||||
#include <string.h>
|
||||
|
||||
#include "opal/util/output.h"
|
||||
#include "orte/mca/oob/oob.h"
|
||||
#include "orte/mca/rml/rml.h"
|
||||
#include "orte/mca/iof/base/iof_base_header.h"
|
||||
#include "orte/mca/iof/base/iof_base_fragment.h"
|
||||
#include "orte/mca/errmgr/errmgr.h"
|
||||
#include "orte/class/orte_proc_table.h"
|
||||
#include "iof_svc.h"
|
||||
#include "iof_svc_proxy.h"
|
||||
#include "iof_svc_pub.h"
|
||||
@ -39,6 +40,8 @@
|
||||
static void orte_iof_svc_sub_construct(orte_iof_svc_sub_t* sub)
|
||||
{
|
||||
sub->sub_endpoint = NULL;
|
||||
sub->has_been_acked = true;
|
||||
sub->last_ack_forwarded = 0;
|
||||
OBJ_CONSTRUCT(&sub->sub_forward, opal_list_t);
|
||||
}
|
||||
|
||||
@ -65,40 +68,45 @@ OBJ_CLASS_INSTANCE(
|
||||
*/
|
||||
|
||||
int orte_iof_svc_sub_create(
|
||||
const orte_process_name_t *src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t *dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
orte_iof_base_tag_t dst_tag)
|
||||
const orte_process_name_t *origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag,
|
||||
const orte_process_name_t *target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
orte_iof_base_tag_t target_tag)
|
||||
{
|
||||
orte_iof_svc_sub_t* sub;
|
||||
opal_list_item_t* item;
|
||||
|
||||
/* See if the subscription already exists */
|
||||
OPAL_THREAD_LOCK(&mca_iof_svc_component.svc_lock);
|
||||
for(item = opal_list_get_first(&mca_iof_svc_component.svc_subscribed);
|
||||
item != opal_list_get_end(&mca_iof_svc_component.svc_subscribed);
|
||||
item = opal_list_get_next(item)) {
|
||||
sub = (orte_iof_svc_sub_t*)item;
|
||||
if (sub->src_mask == src_mask &&
|
||||
orte_ns.compare_fields(sub->src_mask,&sub->src_name,src_name) == 0 &&
|
||||
sub->src_tag == src_tag &&
|
||||
sub->dst_mask == dst_mask &&
|
||||
orte_ns.compare_fields(sub->dst_mask,&sub->dst_name,dst_name) == 0 &&
|
||||
sub->dst_tag == dst_tag) {
|
||||
if (sub->origin_mask == origin_mask &&
|
||||
orte_ns.compare_fields(sub->origin_mask,&sub->origin_name,origin_name) == 0 &&
|
||||
sub->origin_tag == origin_tag &&
|
||||
sub->target_mask == target_mask &&
|
||||
orte_ns.compare_fields(sub->target_mask,&sub->target_name,target_name) == 0 &&
|
||||
sub->target_tag == target_tag) {
|
||||
OPAL_THREAD_UNLOCK(&mca_iof_svc_component.svc_lock);
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
}
|
||||
|
||||
/* No, it does not -- create a new one */
|
||||
sub = OBJ_NEW(orte_iof_svc_sub_t);
|
||||
sub->src_name = *src_name;
|
||||
sub->src_mask = src_mask;
|
||||
sub->src_tag = src_tag;
|
||||
sub->dst_name = *dst_name;
|
||||
sub->dst_mask = dst_mask;
|
||||
sub->dst_tag = dst_tag;
|
||||
sub->sub_endpoint = orte_iof_base_endpoint_match(&sub->dst_name, sub->dst_mask, sub->dst_tag);
|
||||
sub->origin_name = *origin_name;
|
||||
sub->origin_mask = origin_mask;
|
||||
sub->origin_tag = origin_tag;
|
||||
sub->target_name = *target_name;
|
||||
sub->target_mask = target_mask;
|
||||
sub->target_tag = target_tag;
|
||||
sub->sub_endpoint = orte_iof_base_endpoint_match(&sub->target_name, sub->target_mask, sub->target_tag);
|
||||
opal_output(orte_iof_base.iof_output, "created svc sub, origin [%lu,%lu,%lu] tag %d / mask %x, target [%lu,%lu,%lu], tag %d / mask %x\n",
|
||||
ORTE_NAME_ARGS(origin_name), origin_tag, origin_mask,
|
||||
ORTE_NAME_ARGS(target_name), target_tag, target_mask);
|
||||
|
||||
/* search through published endpoints for a match */
|
||||
for(item = opal_list_get_first(&mca_iof_svc_component.svc_published);
|
||||
@ -115,17 +123,231 @@ int orte_iof_svc_sub_create(
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Release resources when the forwarding of an ACK has completed.
|
||||
*/
|
||||
|
||||
static void ack_send_cb(
|
||||
int status,
|
||||
orte_process_name_t* peer,
|
||||
struct iovec* msg,
|
||||
int count,
|
||||
orte_rml_tag_t tag,
|
||||
void* cbdata)
|
||||
{
|
||||
orte_iof_base_frag_t* frag = (orte_iof_base_frag_t*)cbdata;
|
||||
ORTE_IOF_BASE_FRAG_RETURN(frag);
|
||||
if(status < 0) {
|
||||
ORTE_ERROR_LOG(status);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* We have received an ACK from one of the targets that we previously
|
||||
* forwarded a message to. However, given the one-to-many nature of
|
||||
* IOF forwarding, we don't automatically forward that ACK on to the
|
||||
* origin of the original message. Instead, we wait for *all* the
|
||||
* targets of the original message to reply with the apprpriate ACK,
|
||||
* and *then* we forward the ACK on to the original message's origin.
|
||||
*
|
||||
* In this way, the origin will only broadcast as fast as the slowest
|
||||
* target.
|
||||
*
|
||||
* Think of it this way: this function serves as a clearinghouse for
|
||||
* ACKs. It will only push an ACK upstream to an origin when all the
|
||||
* corresponding targets have ACK'ed.
|
||||
*/
|
||||
|
||||
void orte_iof_svc_sub_ack(
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_msg_header_t* hdr,
|
||||
bool do_close)
|
||||
{
|
||||
opal_list_item_t *s_item;
|
||||
uint32_t seq_min;
|
||||
uint32_t last_ack_forwarded = 0;
|
||||
bool has_been_acked = false;
|
||||
bool first = true;
|
||||
union {
|
||||
uint32_t uval;
|
||||
void *vval;
|
||||
} value;
|
||||
|
||||
opal_output(orte_iof_base.iof_output, "orte_iof_svc_proxy_ack");
|
||||
if (do_close) {
|
||||
opal_output(orte_iof_base.iof_output, "CLOSE ACK!\n");
|
||||
}
|
||||
|
||||
/* for each of the subscriptions that match the origin of the ACK:
|
||||
* (1) find all forwarding entries that match the origin of the ACK
|
||||
* (2) update their sequence number
|
||||
* (3) find the minimum sequence number across all endpoints
|
||||
*/
|
||||
|
||||
OPAL_THREAD_LOCK(&mca_iof_svc_component.svc_lock);
|
||||
for(s_item = opal_list_get_first(&mca_iof_svc_component.svc_subscribed);
|
||||
s_item != opal_list_get_end(&mca_iof_svc_component.svc_subscribed);
|
||||
s_item = opal_list_get_next(s_item)) {
|
||||
|
||||
orte_iof_svc_sub_t* sub = (orte_iof_svc_sub_t*)s_item;
|
||||
opal_list_item_t *f_item;
|
||||
|
||||
opal_output(orte_iof_base.iof_output, "ack: checking sub origin [%lu,%lu,%lu] tag %d / mask %x, target [%lu,%lu,%lu], tag %d / mask %x\n",
|
||||
ORTE_NAME_ARGS(&sub->origin_name), sub->origin_tag, sub->origin_mask,
|
||||
ORTE_NAME_ARGS(&sub->target_name), sub->target_tag, sub->target_mask);
|
||||
|
||||
/* If the subscription origin/tag doesn't match the ACK
|
||||
origin/tag, skip it */
|
||||
if (orte_ns.compare_fields(sub->origin_mask,&sub->origin_name,&hdr->msg_origin) != 0 ||
|
||||
sub->origin_tag != hdr->msg_tag) {
|
||||
continue;
|
||||
}
|
||||
|
||||
/* We match, so keep a running tally of whether the ACK has
|
||||
been forwarded or not, and if so, how many bytes have been
|
||||
ACK'ed. */
|
||||
has_been_acked |= sub->has_been_acked;
|
||||
if (sub->has_been_acked) {
|
||||
if (last_ack_forwarded > sub->last_ack_forwarded) {
|
||||
last_ack_forwarded = sub->last_ack_forwarded;
|
||||
}
|
||||
}
|
||||
|
||||
/* Find the minimum amount ack'ed by all the origins (or,
|
||||
technically speaking, ack'ed by their proxies on their
|
||||
behalf) */
|
||||
for(f_item = opal_list_get_first(&sub->sub_forward);
|
||||
f_item != opal_list_get_end(&sub->sub_forward);
|
||||
f_item = opal_list_get_next(f_item)) {
|
||||
orte_iof_svc_fwd_t* fwd = (orte_iof_svc_fwd_t*)f_item;
|
||||
orte_iof_svc_pub_t* pub = fwd->fwd_pub;
|
||||
bool value_set = true;
|
||||
|
||||
opal_output(orte_iof_base.iof_output, "ack: checking fwd [%lu,%lu,%lu] tag %d / mask %x\n",
|
||||
ORTE_NAME_ARGS(&pub->pub_name), pub->pub_tag, pub->pub_mask);
|
||||
|
||||
/* If the publication origin or publication proxy matches
|
||||
the ACK'ing proxy, save the ACK'ed byte count for this
|
||||
*origin* (not the proxy). */
|
||||
if (orte_ns.compare_fields(pub->pub_mask,&pub->pub_name,peer) == 0 ||
|
||||
orte_ns.compare_fields(ORTE_NS_CMP_ALL,&pub->pub_proxy,peer) == 0) {
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"ack: found matching pub");
|
||||
/* If we're closing, then remove this proc from
|
||||
the table -- we won't be using its value to
|
||||
calculate seq_min anymore. Otherwise, put its
|
||||
updated value in the table. */
|
||||
if (do_close) {
|
||||
orte_hash_table_remove_proc(&fwd->fwd_seq_hash,
|
||||
&hdr->msg_origin);
|
||||
value_set = false;
|
||||
} else {
|
||||
value.uval = hdr->msg_seq + hdr->msg_len;
|
||||
orte_hash_table_set_proc(&fwd->fwd_seq_hash,
|
||||
&hdr->msg_origin, &value.vval);
|
||||
}
|
||||
}
|
||||
/* Otherwise, if the publication origin and publication
|
||||
proxy do not match the ACK'ing proxy, then lookup
|
||||
whatever byte count was previously ACK'ed for the origin
|
||||
and use that to compute the minimum byte count ACK'ed
|
||||
so far.
|
||||
|
||||
As such, even though the logic is confusing, at the end
|
||||
of this loop, seq_min will have the minimum number of
|
||||
bytes ACK'ed across all the forwards on this
|
||||
subscription. */
|
||||
else {
|
||||
value.vval = orte_hash_table_get_proc(&fwd->fwd_seq_hash,
|
||||
&hdr->msg_origin);
|
||||
}
|
||||
|
||||
/* If we got a valid value, update the seq_min calculation */
|
||||
if (value_set) {
|
||||
/* If this is the first time we got a valid value,
|
||||
just set it */
|
||||
if (first) {
|
||||
seq_min = value.uval;
|
||||
first = false;
|
||||
}
|
||||
/* Otherwise, only set it if it's a new minimum */
|
||||
else {
|
||||
if (value.uval < seq_min) {
|
||||
seq_min = value.uval;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
OPAL_THREAD_UNLOCK(&mca_iof_svc_component.svc_lock);
|
||||
|
||||
/* If we're closing and there's no longer anyone subscribed,
|
||||
then we're done. */
|
||||
|
||||
if (close && first) {
|
||||
return;
|
||||
}
|
||||
|
||||
/* If everyone has ACK'ed, then push the ACK up to the original
|
||||
message's proxy */
|
||||
if(seq_min == hdr->msg_seq+hdr->msg_len) {
|
||||
/* If the original message was initiated from this process,
|
||||
then the ACK delivery is local. */
|
||||
if(orte_ns.compare_fields(ORTE_NS_CMP_ALL,orte_process_info.my_name,&hdr->msg_origin) == 0) {
|
||||
orte_iof_base_endpoint_t* endpoint;
|
||||
endpoint = orte_iof_base_endpoint_match(&hdr->msg_origin, ORTE_NS_CMP_ALL, hdr->msg_tag);
|
||||
if(endpoint != NULL) {
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"ack: forwarding ack locally: %u", seq_min);
|
||||
orte_iof_base_endpoint_ack(endpoint, seq_min);
|
||||
OBJ_RELEASE(endpoint);
|
||||
}
|
||||
}
|
||||
/* Otherwise, the original message was initiated in another
|
||||
process, and we need to forward the ACK to it. */
|
||||
else {
|
||||
orte_iof_base_frag_t* frag;
|
||||
int rc;
|
||||
|
||||
ORTE_IOF_BASE_FRAG_ALLOC(frag,rc);
|
||||
if(NULL == frag) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
return;
|
||||
}
|
||||
|
||||
frag->frag_hdr.hdr_msg = *hdr;
|
||||
frag->frag_iov[0].iov_base = (IOVBASE_TYPE*)&frag->frag_hdr;
|
||||
frag->frag_iov[0].iov_len = sizeof(frag->frag_hdr);
|
||||
ORTE_IOF_BASE_HDR_MSG_HTON(frag->frag_hdr.hdr_msg);
|
||||
|
||||
opal_output(orte_iof_base.iof_output,
|
||||
"ack: forwarding ack remotely: %u", seq_min);
|
||||
rc = orte_rml.send_nb(
|
||||
&hdr->msg_proxy,
|
||||
frag->frag_iov,
|
||||
1,
|
||||
ORTE_RML_TAG_IOF_SVC,
|
||||
0,
|
||||
ack_send_cb,
|
||||
frag);
|
||||
if(rc < 0) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete all matching subscriptions.
|
||||
*/
|
||||
|
||||
int orte_iof_svc_sub_delete(
|
||||
const orte_process_name_t *src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t *dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
orte_iof_base_tag_t dst_tag)
|
||||
const orte_process_name_t *origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag,
|
||||
const orte_process_name_t *target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
orte_iof_base_tag_t target_tag)
|
||||
{
|
||||
opal_list_item_t *item;
|
||||
OPAL_THREAD_LOCK(&mca_iof_svc_component.svc_lock);
|
||||
@ -133,12 +355,12 @@ int orte_iof_svc_sub_delete(
|
||||
while(item != opal_list_get_end(&mca_iof_svc_component.svc_subscribed)) {
|
||||
opal_list_item_t* next = opal_list_get_next(item);
|
||||
orte_iof_svc_sub_t* sub = (orte_iof_svc_sub_t*)item;
|
||||
if (sub->src_mask == src_mask &&
|
||||
orte_ns.compare_fields(sub->src_mask,&sub->src_name,src_name) == 0 &&
|
||||
sub->src_tag == src_tag &&
|
||||
sub->dst_mask == dst_mask &&
|
||||
orte_ns.compare_fields(sub->dst_mask,&sub->dst_name,dst_name) == 0 &&
|
||||
sub->dst_tag == dst_tag) {
|
||||
if (sub->origin_mask == origin_mask &&
|
||||
orte_ns.compare_fields(sub->origin_mask,&sub->origin_name,origin_name) == 0 &&
|
||||
sub->origin_tag == origin_tag &&
|
||||
sub->target_mask == target_mask &&
|
||||
orte_ns.compare_fields(sub->target_mask,&sub->target_name,target_name) == 0 &&
|
||||
sub->target_tag == target_tag) {
|
||||
opal_list_remove_item(&mca_iof_svc_component.svc_subscribed, item);
|
||||
OBJ_RELEASE(item);
|
||||
}
|
||||
@ -158,10 +380,10 @@ int orte_iof_svc_sub_delete_all(
|
||||
while(item != opal_list_get_end(&mca_iof_svc_component.svc_subscribed)) {
|
||||
opal_list_item_t* next = opal_list_get_next(item);
|
||||
orte_iof_svc_sub_t* sub = (orte_iof_svc_sub_t*)item;
|
||||
if ((sub->src_mask == ORTE_NS_CMP_ALL &&
|
||||
orte_ns.compare_fields(ORTE_NS_CMP_ALL,&sub->src_name,name) == 0) ||
|
||||
(sub->dst_mask == ORTE_NS_CMP_ALL &&
|
||||
orte_ns.compare_fields(ORTE_NS_CMP_ALL,&sub->dst_name,name) == 0)) {
|
||||
if ((sub->origin_mask == ORTE_NS_CMP_ALL &&
|
||||
orte_ns.compare_fields(ORTE_NS_CMP_ALL,&sub->origin_name,name) == 0) ||
|
||||
(sub->target_mask == ORTE_NS_CMP_ALL &&
|
||||
orte_ns.compare_fields(ORTE_NS_CMP_ALL,&sub->target_name,name) == 0)) {
|
||||
opal_list_remove_item(&mca_iof_svc_component.svc_subscribed, item);
|
||||
OBJ_RELEASE(item);
|
||||
}
|
||||
@ -176,13 +398,13 @@ int orte_iof_svc_sub_delete_all(
|
||||
/*
|
||||
* Callback on send completion. Release send resources (fragment).
|
||||
*/
|
||||
|
||||
|
||||
static void orte_iof_svc_sub_send_cb(
|
||||
int status,
|
||||
orte_process_name_t* peer,
|
||||
struct iovec* msg,
|
||||
int count,
|
||||
int tag,
|
||||
orte_rml_tag_t tag,
|
||||
void* cbdata)
|
||||
{
|
||||
orte_iof_base_frag_t* frag = (orte_iof_base_frag_t*)cbdata;
|
||||
@ -225,8 +447,8 @@ int orte_iof_svc_sub_forward(
|
||||
frag->frag_iov[1].iov_base = (IOVBASE_TYPE*)frag->frag_data;
|
||||
frag->frag_iov[1].iov_len = frag->frag_len;
|
||||
memcpy(frag->frag_data, data, frag->frag_len);
|
||||
ORTE_IOF_BASE_HDR_MSG_NTOH(frag->frag_hdr.hdr_msg);
|
||||
rc = mca_oob_send_nb(
|
||||
ORTE_IOF_BASE_HDR_MSG_HTON(frag->frag_hdr.hdr_msg);
|
||||
rc = orte_rml.send_nb(
|
||||
&pub->pub_proxy,
|
||||
frag->frag_iov,
|
||||
2,
|
||||
@ -256,15 +478,16 @@ int orte_iof_svc_sub_forward(
|
||||
static void orte_iof_svc_fwd_construct(orte_iof_svc_fwd_t* fwd)
|
||||
{
|
||||
fwd->fwd_pub = NULL;
|
||||
OBJ_CONSTRUCT(&fwd->fwd_seq, opal_hash_table_t);
|
||||
opal_hash_table_init(&fwd->fwd_seq, 256);
|
||||
OBJ_CONSTRUCT(&fwd->fwd_seq_hash, opal_hash_table_t);
|
||||
opal_hash_table_init(&fwd->fwd_seq_hash, 256);
|
||||
}
|
||||
|
||||
static void orte_iof_svc_fwd_destruct(orte_iof_svc_fwd_t* fwd)
|
||||
{
|
||||
if(NULL != fwd->fwd_pub)
|
||||
if(NULL != fwd->fwd_pub) {
|
||||
OBJ_RELEASE(fwd->fwd_pub);
|
||||
OBJ_DESTRUCT(&fwd->fwd_seq);
|
||||
}
|
||||
OBJ_DESTRUCT(&fwd->fwd_seq_hash);
|
||||
}
|
||||
|
||||
|
||||
@ -283,8 +506,8 @@ bool orte_iof_svc_fwd_match(
|
||||
orte_iof_svc_sub_t* sub,
|
||||
orte_iof_svc_pub_t* pub)
|
||||
{
|
||||
if (orte_ns.compare_fields(sub->dst_mask,&sub->dst_name,&pub->pub_name) == 0 &&
|
||||
sub->src_tag == pub->pub_tag) {
|
||||
if (orte_ns.compare_fields(sub->target_mask,&sub->target_name,&pub->pub_name) == 0 &&
|
||||
sub->origin_tag == pub->pub_tag) {
|
||||
return true;
|
||||
} else {
|
||||
return false;
|
||||
@ -306,6 +529,12 @@ int orte_iof_svc_fwd_create(
|
||||
}
|
||||
OBJ_RETAIN(pub);
|
||||
fwd->fwd_pub = pub;
|
||||
opal_output(orte_iof_base.iof_output, "created svc forward, sub origin [%lu,%lu,%lu], tag %d / mask %x, sub target [%lu,%lu,%lu], tag %d / mask %x :::: pub name [%lu,%lu,%lu], tag %d / mask %x\n",
|
||||
ORTE_NAME_ARGS(&sub->origin_name), sub->origin_tag,
|
||||
sub->origin_mask,
|
||||
ORTE_NAME_ARGS(&sub->target_name), sub->target_tag,
|
||||
sub->target_mask,
|
||||
ORTE_NAME_ARGS(&pub->pub_name), pub->pub_tag, pub->pub_mask);
|
||||
opal_list_append(&sub->sub_forward, &fwd->super);
|
||||
return ORTE_SUCCESS;
|
||||
}
|
||||
|
||||
@ -34,7 +34,7 @@ extern "C" {
|
||||
struct orte_iof_svc_fwd_t {
|
||||
opal_list_item_t super;
|
||||
orte_iof_svc_pub_t* fwd_pub;
|
||||
opal_hash_table_t fwd_seq;
|
||||
opal_hash_table_t fwd_seq_hash;
|
||||
};
|
||||
typedef struct orte_iof_svc_fwd_t orte_iof_svc_fwd_t;
|
||||
|
||||
@ -43,14 +43,16 @@ OBJ_CLASS_DECLARATION(orte_iof_svc_fwd_t);
|
||||
|
||||
struct orte_iof_svc_sub_t {
|
||||
opal_list_item_t super;
|
||||
orte_process_name_t src_name;
|
||||
orte_ns_cmp_bitmask_t src_mask;
|
||||
orte_iof_base_tag_t src_tag;
|
||||
orte_process_name_t dst_name;
|
||||
orte_ns_cmp_bitmask_t dst_mask;
|
||||
orte_iof_base_tag_t dst_tag;
|
||||
orte_process_name_t origin_name;
|
||||
orte_ns_cmp_bitmask_t origin_mask;
|
||||
orte_iof_base_tag_t origin_tag;
|
||||
orte_process_name_t target_name;
|
||||
orte_ns_cmp_bitmask_t target_mask;
|
||||
orte_iof_base_tag_t target_tag;
|
||||
orte_iof_base_endpoint_t* sub_endpoint;
|
||||
opal_list_t sub_forward;
|
||||
bool has_been_acked;
|
||||
uint32_t last_ack_forwarded;
|
||||
};
|
||||
typedef struct orte_iof_svc_sub_t orte_iof_svc_sub_t;
|
||||
|
||||
@ -62,7 +64,7 @@ OBJ_CLASS_DECLARATION(orte_iof_svc_sub_t);
|
||||
*/
|
||||
|
||||
orte_iof_svc_sub_t* orte_iof_svc_sub_lookup(
|
||||
const orte_process_name_t* src
|
||||
const orte_process_name_t* origin
|
||||
);
|
||||
|
||||
/**
|
||||
@ -70,27 +72,36 @@ orte_iof_svc_sub_t* orte_iof_svc_sub_lookup(
|
||||
*/
|
||||
|
||||
int orte_iof_svc_sub_create(
|
||||
const orte_process_name_t *src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t *dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
orte_iof_base_tag_t dst_tag);
|
||||
const orte_process_name_t *origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag,
|
||||
const orte_process_name_t *target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
orte_iof_base_tag_t target_tag);
|
||||
|
||||
/**
|
||||
* Receive an ACK from a single endpoint on a subscription
|
||||
*/
|
||||
|
||||
void orte_iof_svc_sub_ack(
|
||||
const orte_process_name_t* peer,
|
||||
orte_iof_base_msg_header_t* hdr,
|
||||
bool do_close);
|
||||
|
||||
/**
|
||||
* Cleanup/remove a subscription
|
||||
*/
|
||||
|
||||
int orte_iof_svc_sub_delete(
|
||||
const orte_process_name_t *src_name,
|
||||
orte_ns_cmp_bitmask_t src_mask,
|
||||
orte_iof_base_tag_t src_tag,
|
||||
const orte_process_name_t *dst_name,
|
||||
orte_ns_cmp_bitmask_t dst_mask,
|
||||
orte_iof_base_tag_t dst_tag);
|
||||
const orte_process_name_t *origin_name,
|
||||
orte_ns_cmp_bitmask_t origin_mask,
|
||||
orte_iof_base_tag_t origin_tag,
|
||||
const orte_process_name_t *target_name,
|
||||
orte_ns_cmp_bitmask_t target_mask,
|
||||
orte_iof_base_tag_t target_tag);
|
||||
|
||||
int orte_iof_svc_sub_delete_all(
|
||||
const orte_process_name_t *dst_name);
|
||||
const orte_process_name_t *target_name);
|
||||
|
||||
/**
|
||||
* Forward message to any endpoints that
|
||||
|
||||
@ -577,16 +577,46 @@ static void odls_default_wait_local_proc(pid_t pid, int status, void* cbdata)
|
||||
goto GOTCHILD;
|
||||
}
|
||||
}
|
||||
/* get here if we didn't find the child, or if the specified child is already
|
||||
* dead. If the latter, then we have a problem as it means we are detecting
|
||||
* it exiting multiple times
|
||||
/* get here if we didn't find the child, or if the specified child
|
||||
* is already dead. If the latter, then we have a problem as it
|
||||
* means we are detecting it exiting multiple times
|
||||
*/
|
||||
opal_output(orte_odls_globals.output, "odls: did not find pid %ld in table!", (long) pid);
|
||||
ORTE_ERROR_LOG(ORTE_ERR_NOT_FOUND);
|
||||
opal_condition_signal(&orte_odls_default.cond);
|
||||
OPAL_THREAD_UNLOCK(&orte_odls_default.mutex);
|
||||
return;
|
||||
|
||||
GOTCHILD:
|
||||
/* If this child was the (vpid==0), we hooked it up to orterun's
|
||||
STDIN SOURCE earlier (do not change this without also changing
|
||||
odsl_default_fork_local_proc()). So we have to tell the SOURCE
|
||||
a) that we don't want any more data and b) that it should not
|
||||
expect any more ACKs from this endpoint (so that the svc
|
||||
component can still flush/shut down cleanly).
|
||||
|
||||
Note that the source may have already detected that this
|
||||
process died as part of an OOB/RML exception, but that's ok --
|
||||
its "exception" detection capabilities are not reliable, so we
|
||||
*have* to do this unpublish here, even if it arrives after an
|
||||
exception is detected and handled (in which case this unpublish
|
||||
request will be ignored/discarded. */
|
||||
opal_output(orte_odls_globals.output,
|
||||
"odls: pid %ld corresponds to [%lu,%lu,%lu]\n",
|
||||
(long) pid, ORTE_NAME_ARGS(child->name));
|
||||
if (0 == child->name->vpid) {
|
||||
rc = orte_iof.iof_unpublish(child->name, ORTE_NS_CMP_ALL,
|
||||
ORTE_IOF_STDIN);
|
||||
if (ORTE_SUCCESS != rc) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
/* We can't really abort, so keep going... */
|
||||
}
|
||||
}
|
||||
opal_output(orte_odls_globals.output, "orted sent IOF unpub message!\n");
|
||||
|
||||
/* Note that the svc IOF component will detect an exception on the
|
||||
oob because we're shutting it down, so it will take care of
|
||||
closing down any streams that it has open to us. */
|
||||
orte_iof.iof_flush();
|
||||
|
||||
/* determine the state of this process */
|
||||
@ -704,8 +734,10 @@ static int odls_default_fork_local_proc(
|
||||
default */
|
||||
opts.usepty = OMPI_ENABLE_PTY_SUPPORT;
|
||||
|
||||
/* BWB - Fix post beta. Should setup stdin in orterun and
|
||||
make part of the app_context */
|
||||
/* BWB - Fix post beta. Should setup stdin in orterun and make
|
||||
part of the app_context. Do not change this without also
|
||||
changing the reverse of this in
|
||||
odls_default_wait_local_proc(). */
|
||||
if (child->name->vpid == 0) {
|
||||
opts.connect_stdin = true;
|
||||
} else {
|
||||
|
||||
@ -228,7 +228,6 @@ static void orte_rmgr_urm_wireup_stdin(orte_jobid_t jobid)
|
||||
ORTE_ERROR_LOG(rc);
|
||||
}
|
||||
free(name);
|
||||
|
||||
}
|
||||
|
||||
|
||||
@ -481,6 +480,13 @@ static int orte_rmgr_urm_spawn_job(
|
||||
}
|
||||
free(name); /* done with this */
|
||||
|
||||
#if 0
|
||||
{
|
||||
int i = 0;
|
||||
printf("orte_rmgr_urm_wireup_stdin callback: pid %d\n", getpid());
|
||||
while (i == 0) sleep(5);
|
||||
}
|
||||
#endif
|
||||
/* setup the launch system's stage gate counters and subscriptions */
|
||||
if (ORTE_SUCCESS != (rc = orte_rmgr_base_proc_stage_gate_init(*jobid))) {
|
||||
ORTE_ERROR_LOG(rc);
|
||||
|
||||
Загрузка…
x
Ссылка в новой задаче
Block a user