Submitted by Gleb, reviewed by Rich:

Queue_empty is determined by the reader, and is it's local view.
However, the writer may continue writing to this queue.  The decision
to go on to the next cb_fifo is done in an atomic region, checking the
writer's view.  The writer also "changes it's view" in an atomic
region protected by the same lock.

This commit was SVN r13968.

ompi/class/ompi_fifo.h

@@ -422,6 +422,10 @@ void *ompi_fifo_read_from_tail(ompi_fifo_t *fifo)
         opal_atomic_lock(&(fifo->fifo_lock));
         if(fifo->tail->cb_overflow == true) {
             fifo->tail->cb_overflow = false;
+            if(fifo->head == ((char*)fifo->tail) + fifo->offset) {
+                opal_atomic_unlock(&(fifo->fifo_lock));
+                return return_value;
+            }
             fifo->tail = (ompi_cb_fifo_wrapper_t*)
                 ((char*)fifo->tail->next_fifo_wrapper - fifo->offset);
         }