make test: detect hung tests

Run tests in a forked process with a set of pipes to communicate
keep-alives and overall result. This allows us to detect when e.g.
vpp dies mid-API call causing the test to hang waiting for response
(which will never come since vpp died).

Support setting a (per test case) TIMEOUT make test option to set timeout,
with a default timeout of 120 seconds.

Example - fail the test suite if any test-case fails to finish within
300s:

make TIMEOUT=300 test

Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717
Signed-off-by: Klement Sekera <ksekera@cisco.com>

test/debug.py

@@ -0,0 +1,23 @@
+""" debug utilities """
+
+import os
+import pexpect
+
+gdb_path = '/usr/bin/gdb'
+
+
+def spawn_gdb(binary_path, core_path, logger):
+    if os.path.isfile(gdb_path) and os.access(gdb_path, os.X_OK):
+        # automatically attach gdb
+        gdb_cmdline = "%s %s %s" % (gdb_path, binary_path, core_path)
+        gdb = pexpect.spawn(gdb_cmdline)
+        gdb.interact()
+        try:
+            gdb.terminate(True)
+        except:
+            pass
+        if gdb.isalive():
+            raise Exception("GDB refused to die...")
+    else:
+        logger.error("Debugger '%s' does not exist or is not an "
+                     "executable.." % gdb_path)