Fix intermittent UnitTestTimer failures with CUDA

We have been noticing occasional failures of UnitTestTimer on the
dashboard for CUDA devices. The timer seems to be recording less time
than is actually elapsed.

The problem might be that the CUDA timer actually inserts fences to the
CUDA stream rather than record the current time. Thus, the actual time
might start after some pending operations complete.

To attempt to match the UnitTestTimer measurements closer to wall-clock
time, add a synchronize before starting the timer.

vtkm/cont/Timer.cxx

@@ -7,6 +7,7 @@
 //  the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
 //  PURPOSE.  See the above copyright notice for more information.
 //============================================================================
+#include <vtkm/cont/Algorithm.h>
 #include <vtkm/cont/DeviceAdapter.h>
 #include <vtkm/cont/DeviceAdapterList.h>
 #include <vtkm/cont/Logging.h>
@@ -326,5 +327,11 @@ vtkm::Float64 Timer::GetElapsedTime() const
 
   return functor.ElapsedTime;
 }
+
+void Timer::Synchronize() const
+{
+  vtkm::cont::Algorithm::Synchronize(this->Device);
+}
+
 }
 } // namespace vtkm::cont

vtkm/cont/Timer.h

@@ -73,15 +73,25 @@ public:
   vtkm::Float64 GetElapsedTime() const;
 
   /// Returns the device for which this timer is synchronized. If the device adapter has the same
-  /// id as DeviceAdapterTagAny, then the timer will synchronize all devices.
+  /// id as `DeviceAdapterTagAny`, then the timer will synchronize all devices.
   VTKM_CONT vtkm::cont::DeviceAdapterId GetDevice() const { return this->Device; }
 
+  /// Synchronize the device(s) that this timer is monitoring without starting or stopping the
+  /// timer. This is useful for ensuring that external events are synchronized to this timer.
+  ///
+  /// Note that this method will allways block until the device(s) finish even if the
+  /// `Start`/`Stop` methods do not actually block. For example, the timer for CUDA does not
+  /// actually wait for asynchronous operations to finish. Rather, it inserts a fence and
+  /// records the time as fences are encounted. But regardless, this `Synchronize` method
+  /// will block for the CUDA device.
+  VTKM_CONT void Synchronize() const;
+
 private:
   /// Some timers are ill-defined when copied, so disallow that for all timers.
   VTKM_CONT Timer(const Timer&) = delete;
   VTKM_CONT void operator=(const Timer&) = delete;
 
-  DeviceAdapterId Device;
+  vtkm::cont::DeviceAdapterId Device;
   std::unique_ptr<detail::EnabledDeviceTimerImpls> Internal;
 };
 }

vtkm/cont/testing/UnitTestTimer.cxx

@@ -60,6 +60,12 @@ void CheckTime(const vtkm::cont::Timer& timer, vtkm::Float64 expectedTime)
 
 void DoTimerCheck(vtkm::cont::Timer& timer)
 {
+  // Before starting the timer, synchronize the device. Some timers do not record
+  // the start time as the time `Start` is called. Rather, if operations are still
+  // pending on the device, the timer will start recording after those operations
+  // complete. To make sure there are no pending operations, call `Synchronize`.
+  timer.Synchronize();
+
   std::cout << "  Starting timer\n";
   timer.Start();
   VTKM_TEST_ASSERT(timer.Started(), "Timer fails to track started status");