d4afb16f5a
This commit rewrites both the mpool and rcache frameworks. Summary of changes: - Before this change a significant portion of the rcache functionality lived in mpool components. This meant that it was impossible to add a new memory pool to use with rdma networks (ugni, openib, etc) without duplicating the functionality of an existing mpool component. All the registration functionality has been removed from the mpool and placed in the rcache framework. - All registration cache mpools components (udreg, grdma, gpusm, rgpusm) have been changed to rcache components. rcaches are allocated and released in the same way mpool components were. - It is now valid to pass NULL as the resources argument when creating an rcache. At this time the gpusm and rgpusm components support this. All other rcache components require non-NULL resources. - A new mpool component has been added: hugepage. This component supports huge page allocations on linux. - Memory pools are now allocated using "hints". Each mpool component is queried with the hints and returns a priority. The current hints supported are NULL (uses posix_memalign/malloc), page_size=x (huge page mpool), and mpool=x. - The sm mpool has been moved to common/sm. This reflects that the sm mpool is specialized and not meant for any general allocations. This mpool may be moved back into the mpool framework if there is any objection. - The opal_free_list_init arguments have been updated. The unused0 argument is not used to pass in the registration cache module. The mpool registration flags are now rcache registration flags. - All components have been updated to make use of the new framework interfaces. As this commit makes significant changes to both the mpool and rcache frameworks both versions have been bumped to 3.0.0. Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
213 строки
8.0 KiB
Plaintext
213 строки
8.0 KiB
Plaintext
# -*- text -*-
|
|
#
|
|
# Copyright (c) 2011-2015 NVIDIA. All rights reserved.
|
|
# Copyright (c) 2015 Cisco Systems, Inc. All rights reserved.
|
|
# $COPYRIGHT$
|
|
#
|
|
# Additional copyrights may follow
|
|
#
|
|
# $HEADER$
|
|
#
|
|
[cuCtxGetCurrent failed not initialized]
|
|
WARNING: The call to cuCtxGetCurrent() failed while attempting to register
|
|
internal memory with the CUDA environment. The program will continue to run,
|
|
but the performance of GPU memory transfers may be reduced. This failure
|
|
indicates that the CUDA environment is not yet initialized. To eliminate
|
|
this warning, ensure that CUDA is initialized prior to calling MPI_Init.
|
|
|
|
NOTE: You can turn off this warning by setting the MCA parameter
|
|
mpi_common_cuda_warning to 0.
|
|
#
|
|
[cuCtxGetCurrent failed]
|
|
WARNING: The call to cuCtxGetCurrent() failed while attempting to register
|
|
internal memory with the CUDA environment. The program will continue to run,
|
|
but the performance of GPU memory transfers may be reduced.
|
|
cuCtxGetCurrent return value: %d
|
|
|
|
NOTE: You can turn off this warning by setting the MCA parameter
|
|
mpi_common_cuda_warning to 0.
|
|
#
|
|
[cuCtxGetCurrent returned NULL]
|
|
WARNING: The call to cuCtxGetCurrent() failed while attempting to register
|
|
internal memory with the CUDA environment. The program will continue to run,
|
|
but the performance of GPU memory transfers may be reduced. This failure
|
|
indicates that there is no CUDA context yet. To eliminate this warning,
|
|
ensure that there is a CUDA context prior to calling MPI_Init.
|
|
|
|
NOTE: You can turn off this warning by setting the MCA parameter
|
|
mpi_common_cuda_warning to 0.
|
|
#
|
|
[cuMemHostRegister during init failed]
|
|
The call to cuMemHostRegister(%p, %d, 0) failed.
|
|
Host: %s
|
|
cuMemHostRegister return value: %d
|
|
Registration cache: %s
|
|
#
|
|
[cuMemHostRegister failed]
|
|
The call to cuMemHostRegister(%p, %d, 0) failed.
|
|
Host: %s
|
|
cuMemHostRegister return value: %d
|
|
Registration cache: %s
|
|
#
|
|
[cuIpcGetMemHandle failed]
|
|
The call to cuIpcGetMemHandle failed. This means the GPU RDMA protocol
|
|
cannot be used.
|
|
cuIpcGetMemHandle return value: %d
|
|
address: %p
|
|
Check the cuda.h file for what the return value means. Perhaps a reboot
|
|
of the node will clear the problem.
|
|
#
|
|
[cuMemGetAddressRange failed]
|
|
The call to cuMemGetAddressRange failed. This means the GPU RDMA protocol
|
|
cannot be used.
|
|
cuMemGetAddressRange return value: %d
|
|
address: %p
|
|
Check the cuda.h file for what the return value means. Perhaps a reboot
|
|
of the node will clear the problem.
|
|
#
|
|
[cuMemGetAddressRange failed 2]
|
|
The call to cuMemGetAddressRange failed during the GPU RDMA protocol.
|
|
Host: %s
|
|
cuMemGetAddressRange return value: %d
|
|
address: %p
|
|
Check the cuda.h file for what the return value means. This is highly
|
|
unusual and should not happen. The program will probably abort.
|
|
#
|
|
[Out of cuEvent handles]
|
|
The library has exceeded its number of outstanding event handles.
|
|
For better performance, this number should be increased.
|
|
Current maximum handles: %4d
|
|
Suggested new maximum: %4d
|
|
Rerun with --mca mpi_common_cuda_event_max %d
|
|
#
|
|
[cuIpcOpenMemHandle failed]
|
|
The call to cuIpcOpenMemHandle failed. This is an unrecoverable error
|
|
and will cause the program to abort.
|
|
Hostname: %s
|
|
cuIpcOpenMemHandle return value: %d
|
|
address: %p
|
|
Check the cuda.h file for what the return value means. A possible cause
|
|
for this is not enough free device memory. Try to reduce the device
|
|
memory footprint of your application.
|
|
#
|
|
[cuIpcCloseMemHandle failed]
|
|
The call to cuIpcCloseMemHandle failed. This is a warning and the program
|
|
will continue to run.
|
|
cuIpcCloseMemHandle return value: %d
|
|
address: %p
|
|
Check the cuda.h file for what the return value means. Perhaps a reboot
|
|
of the node will clear the problem.
|
|
#
|
|
[cuMemcpyAsync failed]
|
|
The call to cuMemcpyAsync failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
cuMemcpyAsync(%p, %p, %d) returned value %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuEventCreate failed]
|
|
The call to cuEventCreate failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
Hostname: %s
|
|
cuEventCreate return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuEventRecord failed]
|
|
The call to cuEventRecord failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
Hostname: %s
|
|
cuEventRecord return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuEventQuery failed]
|
|
The call to cuEventQuery failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
cuEventQuery return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuIpcGetEventHandle failed]
|
|
The call to cuIpcGetEventHandle failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
cuIpcGetEventHandle return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuIpcOpenEventHandle failed]
|
|
The call to cuIpcOpenEventHandle failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
cuIpcOpenEventHandle return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuStreamWaitEvent failed]
|
|
The call to cuStreamWaitEvent failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
cuStreamWaitEvent return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuEventDestroy failed]
|
|
The call to cuEventDestory failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
cuEventDestory return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuStreamCreate failed]
|
|
The call to cuStreamCreate failed. This is a unrecoverable error and will
|
|
cause the program to abort.
|
|
Hostname: %s
|
|
cuStreamCreate return value: %d
|
|
Check the cuda.h file for what the return vale means.
|
|
#
|
|
[dlopen disabled]
|
|
Open MPI was compiled without dynamic library support (e.g., with the
|
|
--disable-dlopen flag), and therefore cannot utilize CUDA support.
|
|
|
|
If you need CUDA support, reconfigure Open MPI with dynamic library support enabled.
|
|
#
|
|
[dlopen failed]
|
|
The library attempted to open the following supporting CUDA libraries,
|
|
but each of them failed. CUDA-aware support is disabled.
|
|
%s
|
|
If you are not interested in CUDA-aware support, then run with
|
|
--mca mpi_cuda_support 0 to suppress this message. If you are interested
|
|
in CUDA-aware support, then try setting LD_LIBRARY_PATH to the location
|
|
of libcuda.so.1 to get passed this issue.
|
|
#
|
|
[dlsym failed]
|
|
An error occurred while trying to map in the address of a function.
|
|
Function Name: %s
|
|
Error string: %s
|
|
CUDA-aware support is disabled.
|
|
#
|
|
[bufferID failed]
|
|
An error occurred while trying to get the BUFFER_ID of a GPU memory
|
|
region. This could cause incorrect results. Turn of GPU Direct RDMA
|
|
support by running with --mca btl_openib_cuda_want_gdr_support 0.
|
|
Hostname: %s
|
|
cuPointerGetAttribute return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
[cuPointerSetAttribute failed]
|
|
The call to cuPointerSetAttribute with CU_POINTER_ATTRIBUTE_SYNC_MEMOPS
|
|
failed. This is highly unusual and should not happen. The program will
|
|
continue, but report this error to the Open MPI developers.
|
|
Hostname: %s
|
|
cuPointerSetAttribute return value: %d
|
|
Address: %p
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuStreamSynchronize failed]
|
|
The call to cuStreamSynchronize failed. This is highly unusual and should
|
|
not happen. Please report this error to the Open MPI developers.
|
|
Hostname: %s
|
|
cuStreamSynchronize return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[cuMemcpy failed]
|
|
The call to cuMemcpy failed. This is highly unusual and should
|
|
not happen. Please report this error to the Open MPI developers.
|
|
Hostname: %s
|
|
cuMemcpy return value: %d
|
|
Check the cuda.h file for what the return value means.
|
|
#
|
|
[No memory]
|
|
A call to allocate memory within the CUDA support failed. This is
|
|
an unrecoverable error and will cause the program to abort.
|
|
Hostname: %s
|