1
1
openmpi/ompi
Jeff Squyres c8258c06e2 In coll_sm, we alloc a huge chunk of shared memory, divvy it into lots
of individual regions (each region is a multiple of page size in
length), and each process claims its own regions by binding it to its
local memory.  Each process would end up membining something like 16
individual regions in the overall shmem segment.

There were two errors in this code relating to the memory affinity
pinning.  Some combination of these two errors would lead to kernel
panics (!) on my RHEL 6.2 x86_64 machines when used with mmap'ed
shared memory (not posix or sysv shared memory, curiously enough):

1. The shared memory segment is initially divided into two regions:
control and data.  The control starts at the beginning of the shmem
segment, the data starts after that.  The data portion, unfortunately,
was ''not'' aligned to a page.  So all the multiple-of-page-size
regions that we divvy up were also not alined on page boundaries.  And
therefore all the regions we tried to membind were not on page
boundaries.

The solution was to ensure that the data portion started on a page
boundary.  Then all of the individual regions were on page boundaries,
too.

That being said, in my tests, Linux mbind() fails gracefully when the
address is not on a page boundary.  So I'm not sure how this worked at
all / led to a kernel panic...

2. There was some bad pointer math that resulted in membinding regions
larger than they should have been, resulting in region overlaps.
There were definitely overlaps between regions in the same process;
it's likely that there were overlaps between regions of multiple
processes, too -- I'm not sure (and don't care to figure out :-) ).

The solution was to fix the pointer math so that each region membinds
exactly only itself and no neighboring/overlapping regions.

cmr:v1.7.2:reviewer=samuel

This commit was SVN r28442.
2013-05-03 12:49:35 +00:00
..
attribute Implement MPI-2.2 functionality of deleting attributes on 2013-05-02 12:32:21 +00:00
class The constructor and destructor are not publicly visible functions. 2013-04-30 23:23:57 +00:00
communicator Cleanup the communicator cid allocation function. The value of the old_com 2013-03-26 14:45:21 +00:00
contrib Fixed Coverity CID 993726 (Use after free) 2013-04-04 06:57:00 +00:00
datatype Fix the issues with the MPI_Op and the Fortran90 types. 2012-12-19 11:08:18 +00:00
debuggers MCA/base: Add new MCA variable system 2013-03-27 21:09:41 +00:00
errhandler Ensure that MPI_LASTUSEDCODE is always >= MPI_ERR_LASTCODE. Thanks to 2013-03-27 01:32:27 +00:00
etc Use MKDIR_P instead of mkdir_p in Makefiles, as MKDIR_P is the only one 2012-06-21 16:52:37 +00:00
file == Highlights == 2012-04-18 15:57:29 +00:00
group == Highlights == 2012-04-18 15:57:29 +00:00
include Per RFC add initial support for the MPI 3.0 tools interface. 2013-04-24 15:59:23 +00:00
info Update to r27292: only use "valuelen-1" when the valuelen is 2012-10-30 17:56:28 +00:00
mca In coll_sm, we alloc a huge chunk of shared memory, divvy it into lots 2013-05-03 12:49:35 +00:00
message Remove a bunch of unused variables. 2013-03-26 14:34:29 +00:00
mpi Correctly handle the invalid status for null and inactive 2013-05-01 12:55:24 +00:00
mpiext Move the RTE framework change into the trunk. With this change, all non-CR 2013-01-27 23:25:10 +00:00
mpit Per RFC add initial support for the MPI 3.0 tools interface. 2013-04-24 15:59:23 +00:00
op Warning squash: explicitly ignore the return value from asprintf. 2012-06-27 17:40:24 +00:00
patterns MCA/base: Add new MCA variable system 2013-03-27 21:09:41 +00:00
peruse - Sanity check initialization and finalization of PERUSE. 2010-01-12 16:36:24 +00:00
proc In talking about this with Jeff and Ralph, we don't actually need 2013-02-12 21:10:11 +00:00
request Correctly handle the invalid status for null and inactive 2013-05-01 12:55:24 +00:00
runtime Fix typo in check for mpi_leave_pinned vs mpi_leave_pinned_pipeline. 2013-04-30 20:08:32 +00:00
tools Add the ability to use an external version of libevent. Clearly not recommended at this time. I've verified that it works in limited scenarios, but more thorough testing and performance impacts need to be assessed. 2013-04-29 17:02:37 +00:00
win == Highlights == 2012-04-18 15:57:29 +00:00
Makefile.am Per RFC add initial support for the MPI 3.0 tools interface. 2013-04-24 15:59:23 +00:00