This commit adds the flag VTKm_ENABLE_GPU_MPI which when enable it
will use GPU AWARE MPI.
- This will only work with GPUs and MPI implementation that supports GPU
AWARE MPI calls.
- Enabling VTKm_ENABLE_GPU_MPI without MPI/GPU support might results in
errors when running VTK-m with DIY/MPI.
- Only the following tests can run with this feature if enabled:
- UnitTestSerializationDataSet
- UnitTestSerializationArrayHandle
It is possible to create an `ArrayHandleBasic` from a `std::vector`. It
is possible to move the `vector` into the `ArrayHandle` such that the
`ArrayHandle` takes over the memory. When you do this, the `ArrayHandle`
should be able to resize the data if necessary. However, this was not
working.
There were actually 3 problems that were colluding to lead to this
incorrect behavior.
1. The `Buffer` object was not allowing the reallocation of pinned data.
Pinned data means that the `Buffer`'s memory is pointing to some user
data that should stay where it is. Instead, the `Buffer` should attempt
to reallocate the pinned data using its registered realloc method. This
registered realloc method should be the think to throw the exception if
reallocation is not supported. (Technically, the memory doesn't really
need to be pinned since the data is moved and the user no longer has
direct access to it. But for implementation reasons, moved `vector` data
is pinned.)
2. The `InvalidRealloc` function was not properly throwing an exception.
This was not noticed since `Buffer` was inappropriately throwing an
exception for it.
3. The reallocation method `StdVectorReallocater` had a bad assert that
crashed the program during reallocation.
Often when a user gives memory to an `ArrayHandle`, she wants data to be
written into the memory given to be used elsewhere. Previously, the
`Buffer` objects would delete the given buffer as soon as a write buffer
was created elsewhere. That was a problem if a user wants VTK-m to write
results right into a given buffer.
Instead, when a user provides memory, "pin" that memory so that the
`ArrayHandle` never deletes it.
The buffer class encapsulates the movement of raw C arrays between
host and devices.
The `Buffer` class itself is not associated with any device. Instead,
`Buffer` is used in conjunction with a new templated class named
`DeviceAdapterMemoryManager` that can allocate data on a given
device and transfer data as necessary. `DeviceAdapterMemoryManager`
will eventually replace the more complicated device adapter classes
that manage data on a device.
The code in `DeviceAdapterMemoryManager` is actually enclosed in
virtual methods. This allows us to limit the number of classes that
need to be compiled for a device. Rather, the implementation of
`DeviceAdapterMemoryManager` is compiled once with whatever compiler
is necessary, and then the `RuntimeDeviceInformation` is used to
get the correct object instance.
