DOC ONLY: add packet handoff doc

Change-Id: I2e8076bb4f697819780e61ff761defdc74bf4f09
Signed-off-by: Dave Barach <dave@barachs.net>

docs/gettingstarted/developers/vlib.md

@@ -494,3 +494,109 @@ code elsewhere to unpack the data and finally print the answer. If a
 certain cli command has the potential to hurt packet processing
 performance by running for too long, do the work incrementally in a
 process node. The client can wait.
+
+Handing off buffers between threads
+-----------------------------------
+
+Vlib includes an easy-to-use mechanism for handing off buffers between
+worker threads. A typical use-case: software ingress flow hashing. At
+a high level, one creates a per-worker-thread queue which sends packets
+to a specific graph node in the indicated worker thread. With the
+queue in hand, enqueue packets to the worker thread of your choice.
+
+### Initialize a handoff queue
+
+Simple enough, call vlib_frame_queue_main_init:
+
+```c
+   main_ptr->frame_queue_index
+       = vlib_frame_queue_main_init (dest_node.index, frame_queue_size);
+```
+
+Frame_queue_size means what it says: the number of frames which may be
+queued. Since frames contain 1...256 packets, frame_queue_size should
+be a reasonably small number (32...64). If the frame queue producer(s)
+are faster than the frame queue consumer(s), congestion will
+occur. Suggest letting the enqueue operator deal with queue
+congestion, as shown in the enqueue example below.
+
+Under the floorboards, vlib_frame_queue_main_init creates an input queue
+for each worker thread.
+
+Please do NOT create frame queues until it's clear that they will be
+used. Although the main dispatch loop is reasonably smart about how
+often it polls the (entire set of) frame queues, polling unused frame
+queues is a waste of clock cycles.
+
+### Hand off packets
+
+The actual handoff mechanics are simple, and integrate nicely with
+a typical graph-node dispatch function:
+
+```c
+    always_inline uword
+    do_handoff_inline (vlib_main_t * vm,
+         	       vlib_node_runtime_t * node, vlib_frame_t * frame,
+    		       int is_ip4, int is_trace)
+    {
+      u32 n_left_from, *from;
+      vlib_buffer_t *bufs[VLIB_FRAME_SIZE], **b;
+      u16 thread_indices [VLIB_FRAME_SIZE];
+      u16 nexts[VLIB_FRAME_SIZE], *next;
+      u32 n_enq;
+      htest_main_t *hmp = &htest_main;
+      int i;
+
+      from = vlib_frame_vector_args (frame);
+      n_left_from = frame->n_vectors;
+
+      vlib_get_buffers (vm, from, bufs, n_left_from);
+      next = nexts;
+      b = bufs;
+
+      /*
+       * Typical frame traversal loop, details vary with
+       * use case. Make sure to set thread_indices[i] with
+       * the desired destination thread index. You may
+       * or may not bother to set next[i].
+       */
+
+      for (i = 0; i < frame->n_vectors; i++)
+        {
+          <snip>
+          /* Pick a thread to handle this packet */
+          thread_indices[i] = f (packet_data_or_whatever);
+          <snip>
+
+          b += 1;
+          next += 1;
+          n_left_from -= 1;
+        }
+
+       /* Enqueue buffers to threads */
+       n_enq =
+        vlib_buffer_enqueue_to_thread (vm, hmp->frame_queue_index,
+                                       from, thread_indices, frame->n_vectors,
+                                       1 /* drop on congestion */);
+       /* Typical counters,
+      if (n_enq < frame->n_vectors)
+        vlib_node_increment_counter (vm, node->node_index,
+    				 XXX_ERROR_CONGESTION_DROP,
+    				 frame->n_vectors - n_enq);
+      vlib_node_increment_counter (vm, node->node_index,
+    			         XXX_ERROR_HANDED_OFF, n_enq);
+      return frame->n_vectors;
+}
+```
+
+Notes about calling vlib_buffer_enqueue_to_thread(...):
+
+* If you pass "drop on congestion" non-zero, all packets in the
+inbound frame will be consumed one way or the other. This is the
+recommended setting.
+
+* In the drop-on-congestion case, please don't try to "help" in the
+enqueue node by freeing dropped packets, or by pushing them to
+"error-drop." Either of those actions would be a severe error.
+
+* It's perfectly OK to enqueue packets to the current thread.