eBPF in Production: Low-Overhead Linux Observability, Tracing, and Kernel Profiling

As systems engineers, we exist in a state of perpetual trade-off. We need deep, granular visibility into our production systems to debug performance regressions, identify security threats, and ensure reliability. Yet, the very act of observing often perturbs the system we're trying to measure. Traditional tools like strace can impose a crippling 100x performance penalty. Sidecar proxies, the cornerstone of many service meshes, add network latency and consume significant CPU and memory resources for every single pod. This is the "observability tax" - a price we pay in performance and complexity for insight. What if we could achieve near-zero overhead observability by safely programming the kernel itself? This isn't a hypothetical; it's the production reality enabled by eBPF.

What is eBPF? A Kernel-Level Virtual Machine

eBPF (extended Berkeley Packet Filter) is a revolutionary technology that allows us to run sandboxed programs inside the Linux kernel without changing kernel source code or loading kernel modules. Think of it as a small, hyper-efficient, event-driven virtual machine embedded within the kernel. When specific events occur (e.g., a system call is made, a network packet is received, a kernel function is entered), the kernel can execute a pre-loaded eBPF program to collect data, modify behavior, or enforce policy.

This architecture is built on four core components:

1. eBPF Programs: These are small programs written in a restricted subset of C. They are compiled into eBPF bytecode using a compiler like Clang/LLVM.
2. Hooks (Attachment Points): These are well-defined points in the kernel's code path where an eBPF program can be attached. Major types include kprobes (dynamic kernel function instrumentation), tracepoints (stable kernel trace events), XDP (eXpress Data Path for networking), and TC (Traffic Control hooks).
3. eBPF Maps: A set of key-value data structures residing in kernel memory. Maps are the primary mechanism for eBPF programs to store state and to communicate data with userspace applications (and with other eBPF programs). There are various map types, like hash maps, arrays, and high-performance ring buffers.
4. The Verifier: This is the security centerpiece of eBPF. Before any eBPF bytecode is loaded, the verifier performs a static analysis to ensure the program is safe to run. It guarantees that the program will always terminate (no infinite loops), has no out-of-bounds memory access, and cannot cause a kernel panic. If a program fails verification, it is rejected outright.

Once verified, the eBPF bytecode is Just-In-Time (JIT) compiled into native machine code for the host architecture, allowing it to run with near-native performance.

The eBPF Development and Execution Flow

The journey from C code to kernel execution involves a clear, multi-stage process. A userspace "loader" application is responsible for orchestrating this process.

This separation of concerns is critical: the highly privileged kernel executes only verified, safe code, while the complex logic for loading the program and processing the collected data remains in the unprivileged userspace.

Hands-On: Profiling openat System Call Latency

Let's build a practical tool to demonstrate eBPF's power. Our goal is to measure the latency of the openat system call, a common source of I/O wait time. We will record the PID, command, filename, and latency for each call.

We will accomplish this by attaching eBPF programs to two tracepoints: sys_enter_openat and sys_exit_openat.

The eBPF Kernel Program (C)

The C program contains the logic that will run inside the kernel. We'll use a BPF_HASH_MAP to store the start timestamp when a process enters the syscall and a BPF_RINGBUF to send the final event data to userspace. The ring buffer is the modern, preferred way to send high-volume data as it's memory-efficient and doesn't drop data under load, unlike its predecessor, the perf buffer.

// File: syscall_tracer.c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

#define MAX_FILENAME_LEN 256

// Event structure sent to userspace
struct event {
    u32 pid;
    u64 latency_ns;
    char comm[16];
    char filename[MAX_FILENAME_LEN];
};

// Map to store start timestamps, keyed by thread ID (TID)
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);
    __type(value, u64);
} start SEC(".maps");

// Ring buffer to send events to userspace
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024); // 256 KB
} events SEC(".maps");

// Attach to the enter tracepoint for the openat syscall
SEC("tracepoint/syscalls/sys_enter_openat")
int tracepoint__syscalls__sys_enter_openat(struct trace_event_raw_sys_enter* ctx) {
    u64 id = bpf_get_current_pid_tgid();
    u32 tid = (u32)id;
    u64 ts = bpf_ktime_get_ns();

    // Store the start timestamp
    bpf_map_update_elem(&start, &tid, &ts, BPF_ANY);
    return 0;
}

// Attach to the exit tracepoint for the openat syscall
SEC("tracepoint/syscalls/sys_exit_openat")
int tracepoint__syscalls__sys_exit_openat(struct trace_event_raw_sys_exit* ctx) {
    u64 id = bpf_get_current_pid_tgid();
    u32 tid = (u32)id;
    u64 *ts_ptr;

    // Retrieve start timestamp
    ts_ptr = bpf_map_lookup_elem(&start, &tid);
    if (ts_ptr == 0) {
        return 0; // Missed the entry event
    }

    u64 latency = bpf_ktime_get_ns() - *ts_ptr;
    bpf_map_delete_elem(&start, &tid);

    // Filter out very short-lived events
    if (latency < 1000) { // 1 microsecond
        return 0;
    }

    // Reserve space on the ring buffer
    struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e) {
        return 0;
    }

    // Populate event data
    u32 pid = id >> 32;
    e->pid = pid;
    e->latency_ns = latency;

    // Get command name
    bpf_get_current_comm(&e->comm, sizeof(e->comm));

    // Get filename from the context (ctx->args[1] is the char *pathname)
    // Using bpf_probe_read_user_str for safe access to userspace memory
    const char __user *filename_ptr = (const char __user *)ctx->args[1];
    bpf_probe_read_user_str(&e->filename, sizeof(e->filename), filename_ptr);

    // Submit the event to userspace
    bpf_ringbuf_submit(e, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Userspace Loader 1: Python with BCC

The BPF Compiler Collection (BCC) is a toolkit for creating eBPF programs, particularly suited for scripting and rapid prototyping. It embeds the Clang/LLVM compiler and uses a Python (or Lua) frontend to orchestrate the process at runtime.

Here’s how you would load and interact with our C program using BCC:

#!/usr/bin/python3
# File: bcc_loader.py
from bcc import BPF
import ctypes as ct

# The C code from syscall_tracer.c is embedded as a multi-line string
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <linux/fs.h>
#include <linux/sched.h>

#define MAX_FILENAME_LEN 256

struct event {
    u32 pid;
    u64 latency_ns;
    char comm[TASK_COMM_LEN];
    char filename[MAX_FILENAME_LEN];
};
BPF_RINGBUF_OUTPUT(events, 8);

BPF_HASH(start, u32);

int tracepoint__syscalls__sys_enter_openat(struct tracepoint__syscalls__sys_enter_openat *args) {
    u64 id = bpf_get_current_pid_tgid();
    u32 tid = (u32)id;
    u64 ts = bpf_ktime_get_ns();
    start.update(&tid, &ts);
    return 0;
}

int tracepoint__syscalls__sys_exit_openat(struct tracepoint__syscalls__sys_exit_openat *args) {
    u64 id = bpf_get_current_pid_tgid();
    u32 tid = (u32)id;
    u64 *ts_ptr = start.lookup(&tid);

    if (ts_ptr == 0) {
        return 0;
    }

    u64 latency = bpf_ktime_get_ns() - *ts_ptr;
    start.delete(&tid);

    if (latency < 1000) {
        return 0;
    }

    struct event *e = events.ringbuf_reserve(sizeof(struct event));
    if (!e) {
        return 0;
    }

    e->pid = id >> 32;
    e->latency_ns = latency;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    
    // The filename is the second argument to openat
    const char __user* filename_ptr = (const char __user*)args->filename;
    bpf_probe_read_user_str(&e->filename, sizeof(e->filename), filename_ptr);

    events.ringbuf_submit(e, 0);
    return 0;
}
"""

# Define the data structure in Python to match the C struct
class Event(ct.Structure):
    _fields_ = [
        ("pid", ct.c_uint),
        ("latency_ns", ct.c_ulonglong),
        ("comm", ct.c_char * 16),
        ("filename", ct.c_char * 256),
    ]

def print_event(cpu, data, size):
    event = ct.cast(data, ct.POINTER(Event)).contents
    print(f"{event.pid:<10} {event.comm.decode():<16} {event.latency_ns / 1000:<10.2f} us   '{event.filename.decode()}'")

# Load the BPF program
b = BPF(text=bpf_text)

# Attach to the ring buffer
b["events"].open_ring_buffer(print_event)

print(f"{'PID':<10} {'COMM':<16} {'LATENCY':<13} {'FILENAME'}")

# Main loop to poll the ring buffer
while True:
    try:
        b.ring_buffer_poll()
    except KeyboardInterrupt:
        exit()

BCC's magic lies in its runtime compilation. It simplifies development but requires Clang/LLVM to be installed on the target machine and introduces a startup delay. For production binaries, a pre-compiled approach is superior.

Userspace Loader 2: Go with cilium/ebpf

For production services, Go combined with the cilium/ebpf library provides a robust, performant, and self-contained solution. This workflow follows the "Compile Once, Run Everywhere" (CO-RE) paradigm. The C code is compiled ahead of time into an ELF object file, which is then embedded into the Go binary.

First, let's set up the build process using go generate.

# Install the bpf2go tool
go install github.com/cilium/ebpf/cmd/bpf2go@latest

# Create a build file
# File: build.go
//go:build ignore

package main

import (
    "log"
    "github.com/cilium/ebpf/cmd/bpf2go"
)

func main() {
    if err := bpf2go.Generate("syscall_bpf", []string{"syscall_tracer.c"}, "-O2 -g -Wall"); err != nil {
        log.Fatalf("Generating eBPF Go bindings: %v", err)
    }
}

Now, run go generate in your project directory. This will compile syscall_tracer.c and create syscall_bpf.go containing the compiled bytecode and Go structs for accessing maps and programs.

go generate

Now, the Go loader application can be written.

// File: go_loader.go
package main

import (
    "bytes"
    "encoding/binary"
    "errors"
    "log"
    "os"
    "os/signal"
    "syscall"

    "github.com/cilium/ebpf/link"
    "github.com/cilium/ebpf/ringbuf"
)

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go syscall_bpf syscall_tracer.c -- -O2 -g -Wall

func main() {
    stopper := make(chan os.Signal, 1)
    signal.Notify(stopper, os.Interrupt, syscall.SIGTERM)

    // Load pre-compiled programs and maps into the kernel.
    objs := syscall_bpfObjects{}
    if err := loadSyscall_bpfObjects(&objs, nil); err != nil {
        log.Fatalf("loading objects: %v", err)
    }
    defer objs.Close()

    // Attach the tracepoint for sys_enter_openat
    enter_tp, err := link.Tracepoint("syscalls", "sys_enter_openat", objs.TracepointSyscallsSysEnterOpenat, nil)
    if err != nil {
        log.Fatalf("attaching enter tracepoint: %v", err)
    }
    defer enter_tp.Close()

    // Attach the tracepoint for sys_exit_openat
    exit_tp, err := link.Tracepoint("syscalls", "sys_exit_openat", objs.TracepointSyscallsSysExitOpenat, nil)
    if err != nil {
        log.Fatalf("attaching exit tracepoint: %v", err)
    }
    defer exit_tp.Close()

    // Open a reader from the ringbuf map
    rd, err := ringbuf.NewReader(objs.Events)
    if err != nil {
        log.Fatalf("opening ringbuf reader: %v", err)
    }
    defer rd.Close()

    go func() {
        <-stopper
        rd.Close()
    }()

    log.Println("Waiting for events..")
    log.Printf("%-10s %-16s %-13s %s\n", "PID", "COMM", "LATENCY (us)", "FILENAME")

    var event syscall_bpfEvent
    for {
        record, err := rd.Read()
        if err != nil {
            if errors.Is(err, ringbuf.ErrClosed) {
                log.Println("Received signal, exiting..")
                return
            }
            log.Printf("reading from reader: %v", err)
            continue
        }

        // Parse the data
        if err := binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event); err != nil {
            log.Printf("parsing ringbuf event: %v", err)
            continue
        }
        
        // Convert null-terminated C strings to Go strings
        comm := string(event.Comm[:bytes.IndexByte(event.Comm[:], 0)])
        filename := string(event.Filename[:bytes.IndexByte(event.Filename[:], 0)])

        log.Printf("%-10d %-16s %-13.2f %s\n", event.Pid, comm, float64(event.LatencyNs)/1000.0, filename)
    }
}

The Go approach is more verbose but results in a single, static binary with no external runtime dependencies, making it ideal for containerized production deployments.

eBPF vs. Sidecars: The Cilium Case Study

The overhead of the observability tax is most acute in service meshes. A traditional mesh like Istio injects a sidecar proxy (Envoy) into every application pod. This proxy intercepts all network traffic to provide L7 routing, policy enforcement, and observability.

The cost is substantial:

1. Latency: Every request must traverse the TCP/IP stack twice: once from the application to the sidecar, and again from the sidecar to the network.
2. Resource Consumption: Each sidecar is another process, consuming its own CPU and memory, which can double the resource footprint of a pod.
3. Complexity: It adds another moving part to manage, update, and secure.

Cilium, a CNI and service mesh solution built on eBPF, eliminates the sidecar for most networking functions.

• At the Socket Layer: eBPF programs attached to socket hooks (cgroup/sock_addr) can inspect and redirect traffic. For HTTP/gRPC-aware policies, Cilium can transparently redirect traffic to a single per-node Envoy proxy instead of a per-pod sidecar, dramatically reducing resource consumption.
• At the Packet Layer: For L3/L4 load balancing and network policy, eBPF programs on the TC (Traffic Control) and XDP (eXpress Data Path) hooks are used. These operate directly on network packets inside the kernel, providing far greater throughput than iptables or IPVS-based solutions.
• Transparent Observability: Because eBPF can see all kernel-level network events, Cilium can generate rich observability data (e.g., HTTP request/response metrics, network flow logs, DNS lookups) for all traffic without requiring any sidecars or application code changes. This is "free" observability, baked in at the platform layer.

By moving this logic from a userspace sidecar into kernel-level eBPF programs, Cilium bypasses entire layers of the networking stack and avoids the context-switching overhead inherent to the sidecar model.

Test Your Knowledge

Conclusion: The Future is Kernel-Programmable

eBPF represents a fundamental paradigm shift in operating system architecture. It moves us from a world of rigid kernel APIs and high-cost userspace agents to one where we can safely and efficiently program the OS kernel itself to meet our specific needs.

For observability, this means we can finally eliminate the performance tax. We can gather deep, granular data on syscalls, memory allocations, and network traffic directly from the source of truth - the kernel - with minimal overhead. For networking and security, it enables a new generation of tools like Cilium that are faster, more efficient, and more integrated than their sidecar-based predecessors.

Actionable Takeaways:

1. Start Exploring with BCC: On your development machine, install BCC and experiment with its rich library of tools to get a feel for what's possible.
2. Profile Your Applications: Use eBPF to identify performance bottlenecks. Trace syscalls, kernel functions, and even userspace function calls to understand where time is being spent.
3. Invest in CO-RE for Production: When building production-grade monitoring, build your tooling with a libbpf-based library in a compiled language like Go or Rust for maximum performance and portability.
4. Re-evaluate Your Stack: Consider how eBPF-native tools like Cilium could simplify your Kubernetes networking and observability stack, reducing cost and complexity.

eBPF is no longer an esoteric, niche technology. It is a production-ready superpower for any engineer serious about building and running high-performance, observable, and secure systems on Linux.

Further Reading

• bpf.io: The official web presence for all things eBPF.
• Brendan Gregg's Blog: A treasure trove of eBPF performance analysis tools and articles.
• Cilium eBPF Documentation: Deep technical dives into eBPF for networking and observability.
• eBPF GitHub Repository: The libbpf reference library.
• cilium/ebpf for Go: The production-grade Go library for eBPF.