This commit adds the flag VTKm_ENABLE_GPU_MPI which when enable it
will use GPU AWARE MPI.
- This will only work with GPUs and MPI implementation that supports GPU
AWARE MPI calls.
- Enabling VTKm_ENABLE_GPU_MPI without MPI/GPU support might results in
errors when running VTK-m with DIY/MPI.
- Only the following tests can run with this feature if enabled:
- UnitTestSerializationDataSet
- UnitTestSerializationArrayHandle
Previously, the number of buffers held by an `ArrayHandle` had to be
determined statically at compile time by the storage. Most of the time
this is fine. However, there are some exceptions where the number of
buffers need to be selected at runtime. For example, the
`ArrayHandleRecombineVec` does not specify the number of components it
uses, and it needed a hack where it stored buffers in the metadata of
another buffer, which is bad.
This change allows the number of buffers to vary at runtime (at least at
construction). The buffers were already managed in a `std::vector`. It
now no longer forces the vector to be a specific size.
`GetNumberOfBuffers` was removed from the `Storage`. Instead, if the
number of buffers was not specified at construction, an allocation of
size 0 is done to create default buffers.
The biggest change is to the interface of the storage object methods,
which now take `std::vector` instead of pointers to `Buffer` objects.
This adds a little hastle in having to copy subsets of this `vector`
when a storage object has multiple sub-arrays. But it does simplify some
of the templating.
It is possible to create an `ArrayHandleBasic` from a `std::vector`. It
is possible to move the `vector` into the `ArrayHandle` such that the
`ArrayHandle` takes over the memory. When you do this, the `ArrayHandle`
should be able to resize the data if necessary. However, this was not
working.
There were actually 3 problems that were colluding to lead to this
incorrect behavior.
1. The `Buffer` object was not allowing the reallocation of pinned data.
Pinned data means that the `Buffer`'s memory is pointing to some user
data that should stay where it is. Instead, the `Buffer` should attempt
to reallocate the pinned data using its registered realloc method. This
registered realloc method should be the think to throw the exception if
reallocation is not supported. (Technically, the memory doesn't really
need to be pinned since the data is moved and the user no longer has
direct access to it. But for implementation reasons, moved `vector` data
is pinned.)
2. The `InvalidRealloc` function was not properly throwing an exception.
This was not noticed since `Buffer` was inappropriately throwing an
exception for it.
3. The reallocation method `StdVectorReallocater` had a bad assert that
crashed the program during reallocation.
Previously, all of the `ArrayGetValues` implementations were templated
functions that had to be built. That meant that any code using them had
to be compiled with a device compiler and create special code for it.
This change uses an `UnknownArrayHandle` to encapsulate the
`ArrayHandle` and call a per-compiled library function. This means that
the code only has to be compiled once.
The primary purpose of `ArrayHandleRecombineVec` is to take arrays
returned from `ArrayExtractComponent` and recombine them again into a
single `ArrayHandle` that has `Vec` values.
Previously you had to create a special virtual object to place in the
`Buffer`'s metadata. This added a lot of difficulty that is probably
unnecessary.
Instead, just have `Buffer` hold an arbitrary object and some simple
functions to copy and delete it. There may be issues with type safety
across translation units, but we'll deal with that when/if we run into
it.
The new method `ArrayHandle::DeepCopy` had the pattern such that the
data in the `this` array was pushed to the provided destination array.
However, this is the opposite pattern used in the equivalent method in
VTK, which takes the data from the provided array and copies it to
`this` array.
So, swap the pattern to match that of VTK. Also, make the method name
more descriptive by renaming it to `DeepCopyFrom`. Hopefully, users will
read that to mean given the `ArrayHandle`, you copy data from the other
provided `ArrayHandle`.
C++ was not resolving the overloads of `ArrayCopyImpl` as expected,
which was causing `ArrayCopy` to sometimes use a less efficient method
for copying.
Also fix an issue with `Buffer::DeepCopy` that caused a deadlock when
copying a buffer that was not actually allocated anywhere (as well as
failing to copy the metadata, which was probably the whole point).
4345fe26b Store the number of bits of a BitField in the Buffer's metadata
da0403be7 Add metadata to Buffer object.
a84891cd3 Update ArrayHandleBitField to new array style with Buffer
Acked-by: Kitware Robot <kwrobot@kitware.com>
Merge-request: !2218
The number of bits in a `BitField` cannot be directly implied from the
size of the buffer (because the buffer gets padded to the nearest sized
word). Thus, the `BitField stored the number of bits in its own
internals.
Unfortunately, that caused issues when passing the `BitField` data
between it and an `ArrayHandleBitField`. If the `ArrayHandleBitField`
resized itself, the `BitField` would not see the new size because it
ignored the new buffer size.
To get around this problem, `BitField` now declares its own
`BufferMetaData` that stores the number of bits. Now, since the number
of bits is stored in the `Buffer` object, it is sufficient to just share
the `Buffer` to synchronize all of the state.
One of the goals of the `Buffer` object is to allow sharing of data
among objects that will interpret the data differently or give a
different interface over the data. However, when sharing only the array,
important metadata can become lost.
Provide a field that can store some custom metadata in the buffer object
so that the rest of the state can follow the buffer object around.
`ArrayHandle::PrepareForOutput` often has to reallocate the array to the
specified size. Previously, this allocation was not happening with the
`Token` that is passed to `PrepareForOutput`. If the `ArrayHandle` is
already attached or enqueued for that `Token`, then the allocation would
deadlock.
You can now pass a `Token` object to `Allocate`, which is what
`PrepareForOutput` does.
Often when a user gives memory to an `ArrayHandle`, she wants data to be
written into the memory given to be used elsewhere. Previously, the
`Buffer` objects would delete the given buffer as soon as a write buffer
was created elsewhere. That was a problem if a user wants VTK-m to write
results right into a given buffer.
Instead, when a user provides memory, "pin" that memory so that the
`ArrayHandle` never deletes it.
The buffer class encapsulates the movement of raw C arrays between
host and devices.
The `Buffer` class itself is not associated with any device. Instead,
`Buffer` is used in conjunction with a new templated class named
`DeviceAdapterMemoryManager` that can allocate data on a given
device and transfer data as necessary. `DeviceAdapterMemoryManager`
will eventually replace the more complicated device adapter classes
that manage data on a device.
The code in `DeviceAdapterMemoryManager` is actually enclosed in
virtual methods. This allows us to limit the number of classes that
need to be compiled for a device. Rather, the implementation of
`DeviceAdapterMemoryManager` is compiled once with whatever compiler
is necessary, and then the `RuntimeDeviceInformation` is used to
get the correct object instance.
