How to Eliminate the Per‑CPU Map in XDP TCP‑Option Parsing

Background

Original XDP‑based TCP‑option parser used a per‑CPU array map to pass the offset between the XDP program and a freplace helper.

Goal

Remove the per‑CPU map and pass the offset as a function argument.

Step 1 – Remove the per‑CPU map

Function signatures were changed to accept __u8 offset (later changed to int) instead of looking it up from the map. The map definition and helper get_buf were removed. The option_parser, __parse_options, and the freplace program signature were updated, and main.go was adjusted accordingly.

diff --git a/ebpf/tcpoptions/tcp.c b/ebpf/tcpoptions/tcp.c
@@ -29,11 +14,12 @@ __check(void *data, void *data_end, int length)
-option_parser(struct xdp_md *xdp)
+option_parser(struct xdp_md *xdp, __u8 offset)
 {
     int ret = 0;
     barrier_var(ret);
+    barrier_var(offset);
     return xdp ? 1 : ret;
 }
@@ -41,23 +27,20 @@ static void
 __parse_options(struct xdp_md *xdp, struct tcphdr *tcph)
 {
     int length = (tcph->doff << 2) - sizeof(struct tcphdr);
-    __u32 *offset = get_buf();
-    if (!offset)
-        return;
+    __u8 offset;
     /* Initialize offset to tcp options part. */
-    *offset = (void *) (tcph + 1) - ctx_ptr(xdp, data);
+    offset = (void *) (tcph + 1) - ctx_ptr(xdp, data);
@@
-    int ret = option_parser(xdp);
+    int ret = option_parser(xdp, offset);
@@
-    *offset += ret;
+    offset += ret;
     length -= ret;
 }
@@ -55,21 +55,6 @@ static volatile const __u32 TARGET_OPVAL_LEN = 0; // including the suffix '\0'
 #define TCPOLEN_MARK                255
-struct {
-    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
-    __type(key, int);
-    __type(value, __u32);
-    __uint(max_entries, 1);
-} buf SEC(".maps");
-
-static __always_inline __u32 *
-get_buf(void)
-{
-    int key = 0;
-
-    return bpf_map_lookup_elem(&buf, &key);
-}

The accompanying main.go was also updated to match the new signatures.

Verifier failure

Running the program yields:

$ sudo ./tcpoptions
Verifier error: load program: invalid argument:
    Validating topt() func#0...
    0: R1=ctx() R2=scalar() R10=fp0
    ; void *data = ctx_ptr(xdp, data) + offset;
    0: (61) r8 = *(u32 *)(r1 +0)          ; R1=ctx() R8_w=pkt(r=0)
    1: (0f) r8 += r2
    math between pkt pointer and register with unbounded min value is not allowed

The verifier rejects arithmetic between a packet pointer and a register whose minimum value is not bounded.

Attempted guard

Adding if (offset < 20) return -1; still triggers the same error because the verifier cannot infer a bounded range for offset.

Final solution

Change the offset parameter type to int and mask it before pointer arithmetic:

static int parse_option(struct xdp_md *xdp, int offset)
{
    void *data_end = ctx_ptr(xdp, data_end);
    void *data = ctx_ptr(xdp, data);
    struct tcp_option *topt;
    __u8 opcode, opsize;

    offset &= 255;          // limit range for verifier
    data += offset;
    if (!__check(data, data_end, 1))
        return -1;

    opcode = *(__u8 *) data;
    data++;
    // ...
}

Using an int lets the verifier track the numeric range, and the bitwise AND constrains the value to 0‑255, satisfying the verifier.

Why it works

The verifier could not deduce the variable’s type because Clang optimized it away.

The explicit mask provides a concrete range without relying on an if that might be optimized away.

Result

After removing the per‑CPU map, adjusting the offset parameter, and applying the range‑limiting mask, the XDP TCP‑option parser loads and runs successfully.

Reference

Source file: https://github.com/Asphaltt/learn-by-example/blob/main/ebpf/tcpoptions/topt.c