Cilium vs Calico with eBPF: Kubernetes Network Throughput, Security & Service Mesh

Kubernetes networking, traditionally reliant on iptables and IPVS, faces inherent limitations in scalability, performance, and observability. The advent of eBPF has fundamentally reshaped this landscape, enabling kernel-level programmability for networking and security. This document provides a deep dive into Cilium and Calico, specifically focusing on their eBPF-powered datapaths, benchmarking their performance, and evaluating their advanced features like transparent encryption and service mesh integration.

eBPF: The Kernel's Programmable Superpower

eBPF allows user-defined programs to run safely within the kernel, triggered by various events like network packet reception, system calls, or tracepoints. For networking, eBPF programs can attach to network interfaces, tc (traffic control) ingress/egress hooks, or socket operations. This enables highly efficient packet processing, policy enforcement, and load balancing without the overhead of context switching to user space or the complexity of iptables rule traversal.

eBPF in Networking: Key Advantages

1. Performance: Direct packet processing in the kernel bypasses the traditional network stack, reducing latency and increasing throughput.
2. Observability: eBPF programs can extract rich telemetry data from the kernel, providing unparalleled visibility into network flows, latency, and application behavior.
3. Security: Fine-grained policy enforcement at the packet level, including identity-aware security and transparent encryption.
4. Programmability: Dynamic modification of network behavior without kernel module recompilation or reboots.

Cilium with eBPF

Cilium was designed from the ground up to leverage eBPF. Its eBPF-based datapath replaces iptables for network policy enforcement, load balancing, and even kube-proxy functionality.

Key Cilium eBPF Features

• eBPF-based Kube-proxy Replacement: Cilium can entirely replace kube-proxy, performing service load balancing directly in eBPF. This eliminates iptables churn and improves performance.
• Identity-aware Network Policy: Policies are enforced based on Kubernetes labels, not IP addresses, providing more robust and dynamic security.
• Transparent Encryption (WireGuard/IPsec): Encrypts pod-to-pod traffic at the network layer using eBPF, without application changes.
• Service Mesh Integration (Envoy): Deep integration with Envoy for L7 policy enforcement and transparent sidecar injection.
• Observability (Hubble): Built-in eBPF-powered observability platform for network flow visualization and monitoring.

Calico with eBPF

Calico, traditionally known for its iptables-based policy engine, introduced an eBPF datapath option to address performance and scalability concerns. While its policy engine remains robust, the eBPF datapath aims to accelerate packet forwarding and policy enforcement.

Key Calico eBPF Features

• eBPF-based Datapath: Accelerates packet forwarding and policy enforcement by offloading these tasks to eBPF programs.
• Kube-proxy Replacement (Partial): Calico's eBPF mode can handle service load balancing for ClusterIP services, but may still rely on iptables for NodePort and ExternalIPs in some configurations.
• Policy Enforcement: Leverages eBPF for faster application of Calico Network Policies.
• IP-in-IP/VXLAN Encapsulation: Supports standard encapsulation methods, with eBPF accelerating the encapsulation/decapsulation process.

Architectural Comparison: Cilium vs Calico (eBPF)

Benchmarking: Throughput and Latency

To objectively compare Cilium and Calico with their eBPF datapaths, we conducted benchmarks focusing on node-to-node TCP and UDP throughput, and packet processing overhead.

Test Setup

• Kubernetes Version: v1.28.x
• Nodes: 3 x m5.xlarge (4 vCPU, 16 GiB RAM) on AWS EC2
• OS: Ubuntu 22.04 LTS
• Tools: iperf3, netperf
• CNI Versions:
  • Cilium: v1.14.x (eBPF kube-proxy replacement enabled)
  • Calico: v3.26.x (eBPF datapath enabled)
  • Baseline: kube-proxy (iptables) + flannel (VXLAN)

Benchmarking Code (iperf3)

We'll use iperf3 for throughput measurements. The following Kubernetes manifests deploy iperf3 client and server pods.

# iperf3-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: iperf3-server
  labels:
    app: iperf3-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: iperf3-server
  template:
    metadata:
      labels:
        app: iperf3-server
    spec:
      containers:
      - name: iperf3-server
        image: networkstatic/iperf3
        command: ["iperf3", "-s"]
        ports:
        - containerPort: 5201
---
apiVersion: v1
kind: Service
metadata:
  name: iperf3-server
spec:
  selector:
    app: iperf3-server
  ports:
    - protocol: TCP
      port: 5201
      targetPort: 5201

# iperf3-client.yaml
apiVersion: v1
kind: Pod
metadata:
  name: iperf3-client
spec:
  containers:
  - name: iperf3-client
    image: networkstatic/iperf3
    command: ["sleep", "3600"] # Keep pod alive for manual testing

Deployment and Execution:

kubectl apply -f iperf3-server.yaml
kubectl apply -f iperf3-client.yaml

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l app=iperf3-server --timeout=300s
kubectl wait --for=condition=ready pod iperf3-client --timeout=300s

# Get server IP (ClusterIP)
SERVER_IP=$(kubectl get svc iperf3-server -o jsonpath='{.spec.clusterIP}')

# Execute TCP throughput test (client to server)
kubectl exec -it iperf3-client -- iperf3 -c $SERVER_IP -P 8 -t 60

# Execute UDP throughput test (client to server, 100Mbps target)
kubectl exec -it iperf3-client -- iperf3 -c $SERVER_IP -u -b 100M -P 8 -t 60

Benchmark Results (Illustrative)

Analysis:

• eBPF Advantage: Both Cilium and Calico with eBPF significantly outperform the iptables-based Flannel setup. This is primarily due to reduced context switching and more efficient packet processing in the kernel.
• Cilium's Edge: Cilium generally shows higher throughput and lower latency. This can be attributed to its eBPF-native design, which allows for more aggressive optimizations like socket-level load balancing and direct packet forwarding without traversing the full network stack. Calico's eBPF datapath, while effective, still integrates with its existing architecture, which might introduce some overhead.
• CPU Usage: eBPF datapaths typically result in lower CPU utilization on the data plane, as less work is offloaded to user space or the traditional kernel stack.

Advanced Features Deep Dive

Socket-Level Load Balancing (Cilium)

Cilium's eBPF-based kube-proxy replacement operates at the socket layer. When a pod initiates a connection to a ClusterIP service, Cilium's eBPF programs intercept the connect() system call. Instead of modifying iptables rules, eBPF directly rewrites the destination IP/port to a healthy backend pod's IP/port before the connection is established. This is significantly more efficient than iptables NAT, especially for high connection rates.

// Simplified pseudo-code for Cilium's eBPF socket-level load balancing
// This program would attach to the cgroup/connect4 hook

SEC("cgroup/connect4")
int bpf_connect4(struct bpf_sock_addr *ctx) {
    // Check if destination is a ClusterIP
    if (is_cluster_ip(ctx->user_ip4)) {
        // Look up service backends in an eBPF map
        struct service_backend *backend = bpf_map_lookup_elem(&service_backends_map, &ctx->user_ip4);
        if (backend) {
            // Rewrite destination IP and port
            ctx->user_ip4 = backend->ip;
            ctx->user_port = backend->port;
            // Optionally, update source IP for DSR (Direct Server Return)
            // ctx->user_src_ip4 = ...
            // ctx->user_src_port = ...
        }
    }
    return BPF_OK;
}

This approach also enables advanced load balancing algorithms like Maglev or consistent hashing directly in the kernel, ensuring better distribution and fewer connection disruptions during backend changes.

Transparent Encryption (WireGuard/IPsec)

Both Cilium and Calico offer transparent encryption for pod-to-pod traffic.

• Cilium (WireGuard/IPsec): Cilium leverages eBPF to integrate WireGuard or IPsec directly into the datapath. WireGuard, in particular, benefits from its modern cryptographic primitives and kernel-native implementation, offering excellent performance. eBPF programs handle the encryption/decryption process as packets traverse the network stack, making it completely transparent to applications.

  yamlCopy

  # Cilium ConfigMap snippet to enable WireGuard encryption
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: cilium-config
    namespace: kube-system
  data:
    enable-wireguard: "true"
    # Optionally, specify WireGuard interface name
    # wireguard-interface: "wg0"
  

• Calico (IPsec): Calico's IPsec implementation typically relies on strongSwan or similar user-space daemons for key exchange and tunnel management, with kernel modules handling the actual encryption/decryption. While effective, it might incur slightly higher overhead compared to Cilium's eBPF-native WireGuard integration.

Proxyless Envoy Service Mesh Integration (Cilium)

Cilium's most compelling advanced feature is its deep integration with Envoy, enabling a "proxyless" or "sidecar-less" service mesh. Instead of injecting an Envoy sidecar into every application pod, Cilium uses eBPF to redirect traffic to a shared Envoy instance (or a dedicated Envoy instance per node) running as a DaemonSet. This shared Envoy then applies L7 policies, metrics, and tracing, and forwards the traffic back to the destination pod.

This architecture significantly reduces resource consumption (no per-pod sidecar overhead), simplifies deployment, and improves performance by avoiding multiple network hops and context switches.

Production Gotchas & Troubleshooting

1. eBPF Kernel Requirements:

   • Gotcha: Running Cilium/Calico eBPF on older kernels (e.g., < 5.4) can lead to missing features, instability, or outright failure. Some advanced features like kube-proxy replacement or WireGuard require specific kernel versions.
   • Fix: Ensure your Kubernetes nodes are running a modern Linux kernel (5.4+ for basic eBPF, 5.10+ for full feature set, 5.16+ for optimal performance). Use uname -r to check. Upgrade your OS or kernel if necessary.

2. kube-proxy Interference (Calico eBPF):

   • Gotcha: When enabling Calico's eBPF datapath, kube-proxy might still be running and interfering with service load balancing, leading to unpredictable routing or policy issues.
   • Fix: Calico's eBPF mode is designed to replace kube-proxy for ClusterIP services. Ensure kube-proxy is disabled or configured not to manage ClusterIP services. For Calico, set BPF_KUBE_PROXY_IPTABLES_ENABLED=false in the calico-node DaemonSet environment variables.

3. MTU Mismatch:

   • Gotcha: Incorrect MTU settings between nodes or within the CNI can cause packet fragmentation, retransmissions, and significant performance degradation, especially with encapsulation (VXLAN, IP-in-IP) or encryption.
   • Fix: Verify the MTU settings on your node interfaces and CNI configuration. For encapsulated networks, the MTU on the underlying interface should be larger than the pod MTU to accommodate the encapsulation overhead (e.g., 1450 for VXLAN over 1500 byte Ethernet). Use ip link show and check CNI logs.

4. eBPF Program Limits:

   • Gotcha: The Linux kernel has limits on the number and size of eBPF programs and maps. In very large clusters with many policies or services, you might hit these limits, leading to program load failures or unexpected behavior.
   • Fix: Monitor kernel logs for eBPF-related errors. Cilium and Calico are generally optimized to manage these limits. If issues arise, consider simplifying network policies, reducing the number of services, or upgrading to a newer kernel with higher eBPF limits. For Cilium, check cilium status for eBPF map usage.

5. Troubleshooting Connectivity with tcpdump / cilium monitor / calicoctl:

   • Gotcha: Traditional tcpdump might not show the full picture when eBPF is actively manipulating packets in the kernel.
   • Fix:
     • Cilium: Use cilium monitor for real-time visibility into eBPF packet processing, policy decisions, and drops. cilium connectivity test is also invaluable.
     • Calico: Use calicoctl get felixconfiguration default -o yaml to inspect eBPF settings. For general network debugging, tcpdump -i any can still be useful, but be aware of eBPF's influence.

Frequently Asked Questions

1. When should I choose Cilium over Calico (or vice-versa) for eBPF?

   • Choose Cilium if: you prioritize bleeding-edge eBPF features, full kube-proxy replacement, transparent L7 policy with Envoy, advanced observability (Hubble), and WireGuard encryption. It's generally preferred for greenfield deployments or when maximum performance and feature richness are critical.
   • Choose Calico if: you have an existing Calico deployment and want to incrementally improve performance with eBPF while retaining Calico's robust policy engine, or if you need strong multi-cloud/hybrid-cloud capabilities (especially with Calico Enterprise). Its eBPF mode is a strong performance upgrade for its existing users.

2. Does eBPF replace iptables entirely?

   • For Cilium, yes, it can almost entirely replace iptables for network policy, service load balancing, and NAT. Some edge cases or specific HostPort configurations might still touch iptables.
   • For Calico with eBPF, it replaces iptables for core pod-to-pod forwarding and ClusterIP service load balancing. However, NodePort, ExternalIPs, and some other features might still rely on iptables rules managed by kube-proxy or Calico itself, depending on the exact configuration.

3. What are the kernel requirements for running eBPF-based CNIs?

   • A Linux kernel version of 5.4 or newer is generally recommended as a baseline for stable eBPF features. For advanced features like kube-proxy replacement, WireGuard, or optimal performance, kernel 5.10 or 5.16+ is often preferred. Always consult the specific CNI's documentation for precise kernel version compatibility.

4. How does eBPF impact network observability?

   • eBPF significantly enhances observability. Programs can capture detailed metadata about every packet and connection directly from the kernel, including source/destination identity (Kubernetes labels), policy decisions, latency, and drops. Tools like Cilium's Hubble leverage this to provide rich, real-time network flow visualization and troubleshooting capabilities that are impossible with traditional iptables or tcpdump.

5. Is transparent encryption (WireGuard/IPsec) production-ready with eBPF CNIs?

   • Yes, both Cilium's WireGuard and Calico's IPsec implementations are production-ready. Cilium's WireGuard, being eBPF-native, often offers superior performance and simpler management due to its modern design and kernel integration. Always ensure proper key management and certificate rotation practices when deploying encryption in production.

Conclusion

The shift to eBPF-powered datapaths in Kubernetes networking represents a significant leap forward in performance, security, and observability. Cilium, with its eBPF-native architecture, consistently demonstrates superior throughput, lower latency, and a richer feature set, including full kube-proxy replacement, transparent L7 policy, and advanced service mesh integration. Calico's eBPF datapath provides a substantial performance upgrade for its users, maintaining its robust policy engine while leveraging eBPF for accelerated forwarding.

For new Kubernetes deployments or those seeking the absolute cutting edge in networking and security, Cilium with eBPF is the clear leader. For existing Calico users looking to boost performance without a full CNI migration, Calico's eBPF mode offers a compelling path forward. Understanding the nuances of their eBPF implementations is crucial for designing high-performance, secure, and observable Kubernetes clusters.