Only close off stdout/stderr from the daemons if we are not debugging the slurm pls and --debug-daemons was not passed.
This commit was SVN r12276.
The following Trac tickets were found above:
Ticket 352 --> https://svn.open-mpi.org/trac/ompi/ticket/352
Fix the persistent daemon problem where it was exiting when a job completed. Problem was that the persistent daemon would order the job daemons to exit. They would then send an 'ack' back to the persistent daemon - but the ack consisted of an echo of the "exit" command, which was recv'd by the wrong listener who treated it as a properly sent cmd....and exited.
This commit was SVN r12243.
This patch will cause a problem for cnos, however, as there we want to specifically tell the backends to be "null". I'm working on that issue.
This commit was SVN r12225.
processes launched locally for the stdio file names. This was causing
the expected files to not exist and bproc_vexecmove_io to fail.
* Clean up a bunch of debugging output in the bproc pls
This commit was SVN r12102.
I have tested on rsh, slurm, bproc, and tm. Bproc continues to have a problem (will be asking for help there).
Gridengine compiles but I cannot test (believe it likely will run).
Poe and xgrid compile to the extent they can without the proper include files.
This commit was SVN r12059.
* Error message in an NSError object is localizedDescription, not
localizedErrorReason. The latter is a decription of how the error
can occur, which is usually nothing in XGrid frameworks.
* Clean up silly error in finding the Kerberos Service Principal
when using Kerberos authenticaion
* Print useful error message when a connection unexpectedly closes,
as this is usually authentication related...
This commit was SVN r11923.
1. PLS finalize was not being called. Now ensure that happens during orte_finalize.
2. Errmgr proxies were sending their messages to the wrong tag - typical cut/paste error.
This commit was SVN r11891.
Allow the POE RAS to be compled for linux as well as AIX.
The POE RAS is really a Loadleveler RAS, and IU now has
a cluster that uses Loadleveler in a Linux environment (BigRed).
This seems to be the only thing we need to do so far to run
Open MPI on BigRed. Yay :)
This commit was SVN r11600.
set to 1 or 0 instead of the user defined number or default (128).
This caused the PLS to deadlock when using '--debug-daemons' with
more than 2 processes. :(
svn blame says that it was broken in r11347
It is *not* a problem on v1.1 or v1.2 branches.
Bug spotted by Tim Mattox and myself.
This commit was SVN r11575.
The following SVN revision numbers were found above:
r11347 --> open-mpi/ompi@f52c10d18e
create a process component which use CreateProcess to spawn the child.
Special care should be taken in order to correctly redirect the stdin,
stdout and stderr of the child process.
This commit was SVN r11405.
- Remove extra NULL argument from rsh module.
This commit was SVN r11377.
The following SVN revision numbers were found above:
r11347 --> open-mpi/ompi@f52c10d18e
- use the OPAL functions for PATH and environment variables
- make all headers C++ friendly
- no unamed structures
- no implicit cast.
Plus a full implementation for the orte_wait functions.
This commit was SVN r11347.
different macros, one for each project. Therefore, now we have OPAL_DECLSPEC,
ORTE_DECLSPEC and OMPI_DECLSPEC. Please use them based on the sub-project.
This commit was SVN r11270.
Other changes:
1. Remove the old xcpu components as they are not functional.
2. Fix a "bug" in orterun whereby we called dump_aborted_procs even when we normally terminated. There is still some kind of bug in this procedure, however, as we appear to be calling the orterun job_state_callback function every time a process terminates (instead of only once when they have all terminated). I'll continue digging into that one.
This will require an autogen/configure, I'm afraid.
This commit was SVN r11228.
Clean up the remainder of the size_t references in the runtime itself. Convert to orte_std_cntr_t wherever it makes sense (only avoid those places where the actual memory size is referenced).
Remove the obsolete oob barrier function (we actually obsoleted it a long time ago - just never bothered to clean it up).
I have done my best to go through all the components and catch everything, even if I couldn't test compile them since I wasn't on that type of system. Still, I cannot guarantee that problems won't show up when you test this on specific systems. Usually, these will just show as "warning: comparison between signed and unsigned" notes which are easily fixed (just change a size_t to orte_std_cntr_t).
In some places, people didn't use size_t, but instead used some other variant (e.g., I found several places with uint32_t). I tried to catch all of them, but...
Once we get all the instances caught and fixed, this should once and for all resolve many of the heterogeneity problems.
This commit was SVN r11204.
Fixed a few very minor compiler complaints in the pls_gridengine_module.c file. ISO C is less forgiving about where variables get declared.
This commit was SVN r11156.
- indent / whitespace cleanup
- don't set --daemon-debug when pls debug is given, as it seems to make
the daemons abort.
This commit was SVN r11113.
The following SVN revision numbers were found above:
r11109 --> open-mpi/ompi@da7df6d257
Added another system-level test function for ORTE that just spins until terminated by a ctrl-c signal.
Modified orterun - added a couple of newlines to the output when abnormally terminating so the prompt always is on a new line.
This commit was SVN r10866.
keep the resulting tm_event_t that is generated because the back-end
TM library actually caches a bunch of stuff on it for internal
processing, and doesn't let go of it until tm_poll().
tm_event_t's are similar to (but slightly different than)
MPI_Requests: you can't do a million MPI_Isend()'s on a single
MPI_Request -- a) you need an array of MPI_Request's to fill and b)
you need to keep them around until all the requests have completed.
This commit was SVN r10820.
using a pty for everything, which drops all buffered data on the floor when
close() is called on the daemon side, meaning EOF has some issues. Instead,
do the same thing we do for other starters that use the fork() pls -- use
a pipe/fifo for stdin and stderr and a pty for stdout. This is good enough
for what we need and avoids most of the issues with ptys.
This commit was SVN r10692.
* num_children should really be an int instead of size_t
since 'size_t' is not signed and num_children can (in rare cases)
drop below 0, and don't want it to roll around to MAX_INT or some
such.
* I figured out that this problem only happened to me because I use
the pls_fork_reap_timeout MCA parameter and thus the only time that
the code in pls_fork_module.c to waitpid is executed is if this is
not set to 0 (I had it set to 1 to give my procs time to exit). I
adjusted the loop from while{...} to do{...}while; so that it is
executed at least once for consistency.
* de-register the SIGCHILD callback for the pid before we attempt
to kill it, so that we don't leave the door open for both the
waitpids (the one in the callback, and the one in this function)
to race to see who can wait on the child.
* Move the 'thread release' to outside the for loop for a bit of an
optimization, and always set the value to 0 since we want to
finish after this function.
* Added a help message for the case when we can't send a kill()
signal to the process. Should never happen, but all is possible
in the wild wild west of HPC.
This commit was SVN r10666.
When we force an application to terminate (via CTRL-C to mpirun)
we send an out-of-band message to the orted to reap its children.
the fork PLS was doing an internal waitpid but never releasing or
updating the information and signaling the condition variable. So
the fork PLS callback for SIGCHLD registered with the event library
and this waitpid are in a bit of a race to 'waitpid' for the children.
Since the PLS callback was the only one that handled the signal properly
when it 'won' then things were great -- as in the normal termination case.
But when it 'lost' -- as in the abnormal termination case -- the orted
never received the proper signal that its children had gone away.
We want to preserve the internal fork PLS callback since it allows
for a timeout while waiting for the child, which the event library
won't do.
This allows both to exist, and behave properly.
This was introduced in r9068.
The ticket is still open since the orted's hang in other situations
still. This is a fix for one of the causes.
This commit was SVN r10662.
The following SVN revision numbers were found above:
r9068 --> open-mpi/ompi@c2c2daa966
/tmp/tm-merge). Validated by RHC. Summary:
- Add --nolocal (and -nolocal) options to orterun
- Make some scalability improvements to the tm pls
This commit was SVN r10651.
with an error status (< 0) then the req buffer is NULL. Put checks around the
OBJ_RELEASE(req) calls so that we don't try to release NULL :/
This commit was SVN r10641.
After seeing the uglyness that is removing directories in the
codebase I decided to push down this to the OPAL by extending the
opal/os_create_dirpath.(c|h) to contain some more functionality.
In this process I renamed 'os_create_dirpath' to 'os_dirpath' since it
is a bit more general now.
Added a few functions to:
- check if an directory is empty
- check to see if the access permissions are set correctly
- destroy the directory at the end of the dirpath
- By using a caller callback function (a la Perl, I believe)
for every file, the caller can have fine grained control over
whether a specific file is deleted or not.
This simplifies things a bit for orte_session_dir_(finalize|cleanup)
as it should no longer contain any of this functionality, but uses
these functions to do the work.
From the external perspective nothing has changed, from the
developer point of view we have some cleaner, more generic code.
This commit was SVN r10640.
1. Changed the RMGR and PLS APIs to add "signal_job" and "signal_proc" entry points. Only the "signal_job" entries are implemented - none of the components have implementations for "signal_proc" at this time. Thus, you can signal all of the procs in a job, but cannot currently signal only one specific proc.
2. Implemented those new API functions in all components except xgrid (Brian will do so very soon). Only the rsh/ssh and fork modules have been tested, however, and only under OS-X.
3. Added signal traps and callback functions for SIGUSR1/2 to orterun/mpirun that catch those signals and call the appropriate commands to propagate them out to all processes in the job.
4. Added a new test directory under the orte branch to (eventually) hold unit and system level tests for just the run-time. Since our test branch of the repository is under restricted access, people working on the RTE were continually developing their own system-level tests - thus making it hard to help diagnose problems. I have moved the more commonly-used functions here, and added one specifically for testing the SIGUSR1/2 functionality.
I will be contacting people directly to seek help with testing the changes on more environments. Other than compile issues, you should see absolutely no change in behavior on any of your systems - this additional functionality is transparent to anyone who does not issue a SIGUSR1/2 to mpirun.
Ralph
This commit was SVN r10258.
of $libdir and $bindir (i.e., was correctly doing local launches, but
was still using $prefix/lib and $prefix/bin for remote launches).
[Re-]Fixes OFED bug 59.
This commit was SVN r10207.
The following SVN revision numbers were found above:
r9930 --> open-mpi/ompi@1d6902296c
option as discussed on the devel-core mailing list. The Big
Difference is that instead of hard-coding the strings "/lib" and
"/bin" in to append to the prefix, we append the basename of the local
libdir and bindir. Hence, if your libdir is $prefix/lib64, we'll
append /lib64 to construct the remote node's LD_LIBRARY_PATH (etc.).
Also appended the orterun.1 man page to include a description of
--prefix, how it is constructed, what it handles / what it does not,
etc.
This commit was SVN r9930.
installation directories) in configure, the files that depend on this
information are not properly rebuilt. If you need this information,
don't setup a -D in the Makefile.am - instead, include
opal/install_dirs.h.
* Use the : option in AC_CONFIG_FILES to avoid needing to expose that
we are playing around with temporary files with our headers to avoid
rebuilding
* Clean up the version file information a bit, and like the install
directory stuff, make sure that there is a dependency so that
ompi_info gets rebuilt properly when a version number changes.
This commit was SVN r9256.
and cwd update functionality. For bproc, we *do* need to change
directories while checking the cwd because argv[0] may be expressed as
a relative path, and therefore needs to be checked from the cwd
expressed in the app context.
This commit was SVN r9084.
argv[0] and the cwd on the target node (i.e., the node where the
executable will be running in all systems except BProc, where the
searches are run on the node where orterun is invoked).
- fork pls now does cwd and argv[0] search in orted
- bproc pls does cwd and argv[0] search in orterun
- cwd behavior slightly different:
- if user specifies a -wdir to orterun, we chdir() to there; if we
can't for some reason, abort
- if user does not specify a -wdir, try to chdir() to the dir where
orterun was invoked. If we can't for some reason (e.g., it
doesn't exist on the target node), then try to chdir($HOME). If
we can't do that, then just live with whatever default directory
we were put in.
This commit was SVN r9068.
- move files out of toplevel include/ and etc/, moving it into the
sub-projects
- rather than including config headers with <project>/include,
have them as <project>
- require all headers to be included with a project prefix, with
the exception of the config headers ({opal,orte,ompi}_config.h
mpi.h, and mpif.h)
This commit was SVN r8985.
The INIT counter is supposed to be adjusted when the processes are mapped - this is now done correctly.
The LAUNCHED counter is supposed to be adjusted when the pls sets the process pid info into the registry and changes the state to LAUNCHED. This could probably be changed to have that function use the set_proc_soh API, but this fixes the problem for now.
Thanks to Brian for finding that the triggers were not being fired.
This commit was SVN r8948.
and path->agent_path so that it's totally clear what these are for
- make a new rsh component param for agent_param (the value from the
MCA param)
- delay the path check for the agent until the component init -- don't
make it fail during open, because the MCA base will print a warning
if a component fails open() (e.g., on clusters without rsh/ssh (!),
this component was failing noisly even though it was
normal/expected)
This commit was SVN r8596.
In case of checking for Shell with --mca pls_rsh_assume_same_shell 0
have the node point to sensible values.
This commit was SVN r8563.
The following SVN revision numbers were found above:
r7664 --> open-mpi/ompi@0629cdc2d7
now take a colon-delimited list of agents (and associated argv). Also
change the default value to "ssh : rsh". Hence, if we run on a
cluster that does not have ssh, we'll fall back to rsh. If we can't
find rsh, then the rsh component will disqualify itself from
selection.
This commit was SVN r8514.
* turns out (duh!) that there was a reason that the <projectdir>dir
variable was set in the AM conditional. If not, stupid directories
are created and not needed... duh.
This commit was SVN r8205.
component/base Makefile.am files, reducing the time configure spends
stamping out Makefiles at the end
* Install base_impl.h file when devel-headers are being installed
This commit was SVN r8200.
originally suggested by Ralf Wildenhues, to try to speed autogen, configure,
and make (and possibly even make install). Use automake's include directive
to drastically reduce the number of Makefile files (although the number of
Makefile.am files is the same - most are just included in a top-level
Makefile.am). Also use an Automake SUBDIRs feature to eliminate the
dynamic-mca tree, which was no longer really needed. This makes adding
a framework easier (since you don't have to remember the dynamic-mca
tree) and makes building faster (as make doesn't have to recurse through
the dynamic-mca tree)
This commit was SVN r7777.
oversubscribed on a node. And thus whether to call sched_yield or not.
The value of node->node_slots_inuse does not currently represent the number of
slots actually in use, at the moment. This is actually a bug in the RAS/RMAPS
base components, but the fix for that specific bug is bigger than we want to
address at the moment (but will certianly do so in the near future).
Since we cannot trust this value, use the total number of mapped processes
(which was properly set by the RMAPS component upon mapping -- Just not
properly propagated back to the registry's node segment) from the process
mapping.
In addition to this change I cleaned up a couple of the debug messages. It
seems that TM and RSH are the only two directly effected by this. SLURM
would be if that section of code wasn't currently inactive, but put the fix
in for prosparity.
This commit was SVN r7743.
too distant past
* work around apparently broken handling of max_slots somewhere along
the line by just setting it to 0
Both changes should go to the trunk.
This commit was SVN r7710.
command:
svn merge -r 7567:7663 https://svn.open-mpi.org/svn/ompi/tmp/jjhursey-rmaps .
(where "." is a trunk checkout)
The logs from this branch are much more descriptive than I will put
here (including a *really* long description from last night). Here's
the short version:
- fixed some broken implementations in ras and rmaps
- "orterun --host ..." now works and has clearly defined semantics
(this was the impetus for the branch and all these fixes -- LANL had
a requirement for --host to work for 1.0)
- there is still a little bit of cleanup left to do post-1.0 (we got
correct functionality for 1.0 -- we did not fix bad implementations
that still "work")
- rds/hostfile and ras/hostfile handshaking
- singleton node segment assignments in stage1
- remove the default hostfile (no need for it anymore with the
localhost ras component)
- clean up pls components to avoid duplicate ras mapping queries
- [possible] -bynode/-byslot being specific to a single app context
This commit was SVN r7664.
it's always ORTE_SUCCESS and sometimes masks real !=ORTE_SUCCESS rc
values.
- Add MCA param pls_tm_want_path_check. If nonzero (the default),
check for the orted in the PATH before each tm_spawn()'ing (doing a
little caching so that we don't hammer on the filesystem -- remember
all the PATH's where we successfully found the orted so that we
don't have to query the filesystem multiple times for a PATH where
we previously found the orted)
- Be sure to opal_argv_split() the pls_tm_orted MCA param
This commit was SVN r7625.
If you use --prefix and then "-x LD_LIBRARY_PATH", the rsh pls would
take great pains to ensure that PATH and LD_LIBRARY_PATH were setup
correctly on the local and remote nodes, but then the fork pls would
blitely overwrite LD_LIBRARY_PATH with what the user exported (i.e.,
most likely without our prefix). This patch takes care of that -- the
fork pls examines the incoming environment, and if it sees PATH or
LD_LIBRARY_PATH, it re-prefixes those variables.
This commit was SVN r7566.
- Fix bug identified by users: --prefix may also apply on the local
node; we need to prefix the PATH and LD_LIBRARY_PATH environment
variables before invoking execve()
This commit was SVN r7541.
add an event, it can call the spawn function directly. This will avoid it standing on the condition who
will never get released.
This commit was SVN r7428.
automagically don't build on platforms without such things
* Fix for mistaken use of cache variable in assembly setup
* one more cached test hits the books
This commit was SVN r7404.
The following formats are parsed:
user@IPv4
user@fqdn
IPv4 or fqdn [username|user-name|user_name]=user
- Try a better error-detection when parsing (recognize wrong
IPs, fqdns...)
This commit was SVN r7288.
AM_INIT_AUTOMAKE, instead of the deprecated version.
* Work around dumbness in modern AC_INIT that requires the version
number to be set at autoconf time (instead of at configure time, as
it was before). Set the version number, minus the subversion r number,
at autoconf time. Override the internal variables to include the r
number (if needed) at configure time. Basically, the right thing
should always happen. The only place it might not is the version
reported as part of configure --help will not have an r number.
* Since AM_INIT_AUTOMAKE taks a list of options, no need to specify
them in all the Makefile.am files.
* Addes support for subdir-objects, meaning that object files are put
in the directory containing source files, even if the Makefile.am is
in another directory. This should start making it feasible to
reduce the number of Makefile.am files we have in the tree, which
will greatly reduce the time to run autogen and configure.
This commit was SVN r7211.
This allows the user to specify certain options to srun when an application
is launched with this PLS.
A useful example is the need to set the time to wait from when the first
process completes and when slurm kills remaining processes:
pls_slurm_args=--wait=1200
This commit was SVN r7206.
app_context:
mpirun -np 2 -prefix /path/to/ompi/on/machineA ./exec1 : \
-np 2 -prefix /path/to/ompi/on/machineB ./exec2
- Allow with -mca pls_rsh_assume_same_shell 0, the checking for the
SHELL-variable on the actual node (currently 1st node).
Sets the prefix, PATH and LD_LIBRARY_PATH for bash/ksh and
csh/tcsh.
This commit was SVN r7195.
add a -I to find the included ltdl.h (vs. a system-installed ltdl.h)
- Clean up kruft in a bunch of Makefile.am's to remove now-unnecessary
AM_CPPFLAGS settings to get static-components.h for each framework
- Move the component_repository API functions out of opal/mca/base/base.h
and into opal/mca/base/mca_base_component_repository.h in order to
decrease unnecessary dependencies (e.g., before this, almost
everything in the tree depended on ltdl.h, which is unnecessary --
only a small number of files really need ltdl.h)
This commit was SVN r7127.
it to be an exit.
* Put the srun process (or what is about to become the srun process) in
it's own process group so that group-wide signals (such as the
SIGINT sent by hitting cntl-c in a shell) are not sent to the srun
process.
This commit was SVN r7068.
tree.
- fix up #include's throughout the tree (yay contrib/search_replace.pl!)
- remove a few extraneous #include's
- remove orte_sys_info*() from opal_init()/opal_finalize() (it's
already in orte_init_stage1() and orte_system_finalize())
- remove dependencies in opal on orte_system_info -- util/os_path.c
and util/os_create_dirpath.c (they only used path_sep, anyway --
easily changed to #defines)
This commit was SVN r7059.
session directory cleanup (among other things)
- When we get an abnormal exit in orterun (i.e., timeout expires and
we haven't gotten termination notices from all processes), print a
better message an exit in a better way (which includes session
directory cleanup)
- Fix tm and poe pls's to not exit() but rather propagate the error up
the stack (where relevant)
This commit was SVN r7058.
