Buffer Metadata

Each vlib_buffer_t (packet buffer) carries buffer metadata which describes the current packet-processing state. The underlying techniques have been used for decades, across multiple packet processing environments.

We will examine vpp buffer metadata in some detail, but folks who need to manipulate and/or extend the scheme should expect to do a certain level of code inspection.

Vlib (Vector library) primary buffer metatdata

The first 64 octets of each vlib_buffer_t carries the primary buffer metadata. See .../src/vlib/buffer.h for full details.

Important fields:

  • i16 current_data: the signed offset in data[], pre_data[] that we are currently processing. If negative current header points into the pre-data (rewrite space) area.
  • u16 current_length: nBytes between current_data and the end of this buffer.
  • u32 flags: Buffer flag bits. Heavily used, not many bits left
    • src/vlib/buffer.h flag bits
      • VLIB_BUFFER_IS_TRACED: buffer is traced
      • VLIB_BUFFER_NEXT_PRESENT: buffer has multiple chunks
      • VLIB_BUFFER_TOTAL_LENGTH_VALID: total_length_not_including_first_buffer is valid (see below)
    • src/vnet/buffer.h flag bits
      • VNET_BUFFER_F_L4_CHECKSUM_COMPUTED: tcp/udp checksum has been computed
      • VNET_BUFFER_F_L4_CHECKSUM_CORRECT: tcp/udp checksum is correct
      • VNET_BUFFER_F_VLAN_2_DEEP: two vlan tags present
      • VNET_BUFFER_F_VLAN_1_DEEP: one vlag tag present
      • VNET_BUFFER_F_SPAN_CLONE: packet has already been cloned (span feature)
      • VNET_BUFFER_F_LOOP_COUNTER_VALID: packet look-up loop count valid
      • VNET_BUFFER_F_LOCALLY_ORIGINATED: packet built by vpp
      • VNET_BUFFER_F_IS_IP4: packet is ipv4, for checksum offload
      • VNET_BUFFER_F_IS_IP6: packet is ipv6, for checksum offload
      • VNET_BUFFER_F_OFFLOAD_IP_CKSUM: hardware ip checksum offload requested
      • VNET_BUFFER_F_OFFLOAD_TCP_CKSUM: hardware tcp checksum offload requested
      • VNET_BUFFER_F_OFFLOAD_UDP_CKSUM: hardware udp checksum offload requested
      • VNET_BUFFER_F_IS_NATED: natted packet, skip input checks
      • VNET_BUFFER_F_L2_HDR_OFFSET_VALID: L2 header offset valid
      • VNET_BUFFER_F_L3_HDR_OFFSET_VALID: L3 header offset valid
      • VNET_BUFFER_F_L4_HDR_OFFSET_VALID: L4 header offset valid
      • VNET_BUFFER_F_FLOW_REPORT: packet is an ipfix packet
      • VNET_BUFFER_F_IS_DVR: packet to be reinjected into the l2 output path
      • VNET_BUFFER_F_QOS_DATA_VALID: QoS data valid in vnet_buffer_opaque2
      • VNET_BUFFER_F_GSO: generic segmentation offload requested
      • VNET_BUFFER_F_AVAIL1: avaliable bit
      • VNET_BUFFER_F_AVAIL2: avaliable bit
      • VNET_BUFFER_F_AVAIL3: avaliable bit
      • VNET_BUFFER_F_AVAIL4: avaliable bit
      • VNET_BUFFER_F_AVAIL5: avaliable bit
      • VNET_BUFFER_F_AVAIL6: avaliable bit
      • VNET_BUFFER_F_AVAIL7: avaliable bit
  • u32 flow_id: generic flow identifier
  • u8 ref_count: buffer reference / clone count (e.g. for span replication)
  • u8 buffer_pool_index: buffer pool index which owns this buffer
  • vlib_error_t (u16) error: error code for buffers enqueued to error handler
  • u32 next_buffer: buffer index of next buffer in chain. Only valid if VLIB_BUFFER_NEXT_PRESENT is set
  • union
    • u32 current_config_index: current index on feature arc
    • u32 punt_reason: reason code once packet punted. Mutually exclusive with current_config_index
  • u32 opaque[10]: primary vnet-layer opaque data (see below)
  • END of first cache line / data initialized by the buffer allocator
  • u32 trace_index: buffer's index in the packet trace subsystem
  • u32 total_length_not_including_first_buffer: see VLIB_BUFFER_TOTAL_LENGTH_VALID above
  • u32 opaque2[14]: secondary vnet-layer opaque data (see below)
  • u8 pre_data[VLIB_BUFFER_PRE_DATA_SIZE]: rewrite space, often used to prepend tunnel encapsulations
  • u8 data[0]: buffer data received from the wire. Ordinarily, hardware devices use b->data[0] as the DMA target but there are exceptions. Do not write code which blindly assumes that packet data starts in b->data[0]. Use vlib_buffer_get_current(...).

Vnet (network stack) primary buffer metadata

Vnet primary buffer metadata occupies space reserved in the vlib opaque field shown above, and has the type name vnet_buffer_opaque_t. Ordinarily accessed using the vnet_buffer(b) macro. See ../src/vnet/buffer.h for full details.

Important fields:

  • u32 sw_if_index[2]: RX and TX interface handles. At the ip lookup stage, vnet_buffer(b)->sw_if_index[VLIB_TX] is interpreted as a FIB index.
  • i16 l2_hdr_offset: offset from b->data[0] of the packet L2 header. Valid only if b->flags & VNET_BUFFER_F_L2_HDR_OFFSET_VALID is set
  • i16 l3_hdr_offset: offset from b->data[0] of the packet L3 header. Valid only if b->flags & VNET_BUFFER_F_L3_HDR_OFFSET_VALID is set
  • i16 l4_hdr_offset: offset from b->data[0] of the packet L4 header. Valid only if b->flags & VNET_BUFFER_F_L4_HDR_OFFSET_VALID is set
  • u8 feature_arc_index: feature arc that the packet is currently traversing
  • union
    • ip
      • u32 adj_index[2]: adjacency from dest IP lookup in [VLIB_TX], adjacency from source ip lookup in [VLIB_RX], set to ~0 until source lookup done
      • union
        • generic fields
        • ICMP fields
        • reassembly fields
    • mpls fields
    • l2 bridging fields, only valid in the L2 path
    • l2tpv3 fields
    • l2 classify fields
    • vnet policer fields
    • MAP fields
    • MAP-T fields
    • ip fragmentation fields
    • COP (whitelist/blacklist filter) fields
    • LISP fields
    • TCP fields
      • connection index
      • sequence numbers
      • header and data offsets
      • data length
      • flags
    • SCTP fields
    • NAT fields
    • u32 unused[6]

Vnet (network stack) secondary buffer metatdata

Vnet primary buffer metadata occupies space reserved in the vlib opaque2 field shown above, and has the type name vnet_buffer_opaque2_t. Ordinarily accessed using the vnet_buffer2(b) macro. See ../src/vnet/buffer.h for full details.

Important fields:

  • qos fields
    • u8 bits
    • u8 source
  • u8 loop_counter: used to detect and report internal forwarding loops
  • group-based policy fields
    • u8 flags
    • u16 sclass: the packet's source class
  • u16 gso_size: L4 payload size, persists all the way to interface-output in case GSO is not enabled
  • u16 gso_l4_hdr_sz: size of the L4 protocol header
  • union
    • packet trajectory tracer (largely deprecated)
      • u16 *trajectory_trace; only #if VLIB_BUFFER_TRACE_TRAJECTORY > 0
    • packet generator
      • u64 pg_replay_timestamp: timestamp for replayed pcap trace packets
    • u32 unused[8]

Buffer Metadata Extensions

Plugin developers may wish to extend either the primary or secondary vnet buffer opaque unions. Please perform a manual live variable analysis, otherwise nodes which use shared buffer metadata space may break things.

It's not OK to add plugin or proprietary metadata to the core vpp engine header files named above. Instead, proceed as follows. The example concerns the vnet primary buffer opaque union vlib_buffer_opaque_t. It's a very simple variation to use the vnet secondary buffer opaque union vlib_buffer_opaque2_t.

In a plugin header file:

    /* Add arbitrary buffer metadata */
    #include <vnet/buffer.h>

    typedef struct
    {
      u32 my_stuff[6];
    } my_buffer_opaque_t;

    STATIC_ASSERT (sizeof (my_buffer_opaque_t) <=
                   STRUCT_SIZE_OF (vnet_buffer_opaque_t, unused),
                   "Custom meta-data too large for vnet_buffer_opaque_t");

    #define my_buffer_opaque(b)  \
      ((my_buffer_opaque_t *)((u8 *)((b)->opaque) + STRUCT_OFFSET_OF (vnet_buffer_opaque_t, unused)))

To set data in the custom buffer opaque type given a vlib_buffer_t *b:

    my_buffer_opaque (b)->my_stuff[2] = 123;

To read data from the custom buffer opaque type:

    stuff0 = my_buffer_opaque (b)->my_stuff[2];