Cycles: Stop rendering when bad_alloc happens

This is an attempt to gracefully handle out-of-memory events
and stop rendering with an error message instead of a crash.

It uses bad_alloc exception, and usually i'm not really fond
of exceptions, but for such limited use for errors from which
we can't recover it should be fine.

Ideally we'll need to stop full Cycles Session, so viewport
render and persistent images frees all the memory, but that
we can support later, since it'll mainly related on telling
Blender what to do.

General rules are:

- Use as less exception handles as possible, try to find a
  most geenric pace where to handle those.

  For example, ccl::Session.

- Threads needs own handling, exception trap from one thread
  will not catch exceptions from other threads.

  That's why BVH build needs own thing.

Reviewers: brecht, juicyfruit, dingto, lukasstockner97

Differential Revision: https://developer.blender.org/D1898

intern/cycles/render/mesh.cpp

@@ -528,7 +528,7 @@ void Mesh::compute_bvh(SceneParams *params, Progress *progress, int n, int total
 
 			delete bvh;
 			bvh = BVH::create(bparams, objects);
-			bvh->build(*progress);
+			MEM_GUARDED_CALL(progress, bvh->build, *progress);
 		}
 	}
 

intern/cycles/render/session.cpp

@@ -816,7 +816,7 @@ void Session::update_scene()
 	/* update scene */
 	if(scene->need_update()) {
 		progress.set_status("Updating Scene");
-		scene->device_update(device, progress);
+		MEM_GUARDED_CALL(&progress, scene->device_update, device, progress);
 	}
 }
 

intern/cycles/util/util_guarded_allocator.h

@@ -164,6 +164,26 @@ public:
 size_t util_guarded_get_mem_used(void);
 size_t util_guarded_get_mem_peak(void);
 
+/* Call given function and keep track if it runs out of memory.
+ *
+ * If it does run out f memory, stop execution and set progress
+ * to do a global cancel.
+ *
+ * It's not fully robust, but good enough to catch obvious issues
+ * when running out of memory.
+ */
+#define MEM_GUARDED_CALL(progress, func, ...) \
+	do { \
+		try { \
+			(func)(__VA_ARGS__); \
+		} \
+		catch (std::bad_alloc&) { \
+			fprintf(stderr, "Error: run out of memory!\n"); \
+			fflush(stderr); \
+			(progress)->set_error("Out of memory"); \
+		} \
+	} while(false)
+
 CCL_NAMESPACE_END
 
 #endif  /* __UTIL_GUARDED_ALLOCATOR_H__ */