Kubernetes Gateway API with Envoy Gateway: Replacing Ingress-NGINX with Modern Traffic Routing

The Kubernetes Ingress API, while foundational, exhibits inherent limitations in expressing advanced traffic management policies, role-based access control, and protocol-aware routing. These constraints often necessitate vendor-specific annotations or custom resource definitions (CRDs), leading to fragmented configurations and reduced portability. The Kubernetes Gateway API emerges as the strategic successor, offering a more expressive, extensible, and role-oriented approach to traffic management.

This guide details the architectural shift from Ingress-NGINX to the Kubernetes Gateway API, specifically leveraging Envoy Gateway. We will cover the core Gateway API resources, demonstrate advanced traffic routing patterns, implement global rate limiting, automate TLS termination with cert-manager, and outline a zero-downtime migration strategy.

Understanding the Kubernetes Gateway API

The Gateway API introduces a structured, hierarchical model for managing ingress traffic, designed to address the shortcomings of the Ingress API. It separates concerns across different personas: infrastructure providers, cluster operators, and application developers.

Core Resources

The Gateway API defines several key resources:

1. GatewayClass:

   • Defines a class of Gateways, analogous to StorageClass for persistent volumes.
   • Specifies the controller responsible for provisioning and managing Gateways of this class (e.g., envoy-gateway).
   • Managed by infrastructure providers.
   yamlCopy

   apiVersion: gateway.networking.k8s.io/v1
   kind: GatewayClass
   metadata:
     name: eg-standard
   spec:
     controllerName: gateway.envoyproxy.io/gatewayclass-controller
     description: Envoy Gateway managed by Envoy Gateway project.
   

2. Gateway:

   • Represents a logical point of entry for traffic into the cluster.
   • Provisions an actual load balancer or proxy (e.g., an Envoy proxy deployment and a LoadBalancer service).
   • Specifies listeners (ports, protocols, hostnames) and references a GatewayClass.
   • Managed by cluster operators.
   yamlCopy

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: eg-gateway
     namespace: envoy-gateway-system
   spec:
     gatewayClassName: eg-standard
     listeners:
       - name: http
         protocol: HTTP
         port: 80
         allowedRoutes:
           namespaces:
             from: All
       - name: https
         protocol: HTTPS
         port: 443
         tls:
           mode: Terminate
           certificateRefs:
             - kind: Secret
               name: example-com-tls
         allowedRoutes:
           namespaces:
             from: All
   

3. HTTPRoute:

   • Defines rules for routing HTTP/HTTPS traffic from a Gateway to backend services.
   • Supports host, path, header, and query parameter matching.
   • Enables advanced features like weighted traffic splitting, redirects, and rewrites.
   • Managed by application developers.
   yamlCopy

   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: example-httproute
     namespace: default
   spec:
     parentRefs:
       - name: eg-gateway
         namespace: envoy-gateway-system
     hostnames:
       - "example.com"
     rules:
       - matches:
           - path:
               type: PathPrefix
               value: /api
         backendRefs:
           - name: my-service
             port: 8080
   

4. GRPCRoute:

   • Specialized route for gRPC traffic, allowing matching on gRPC service and method names.
   • Managed by application developers.
   yamlCopy

   apiVersion: gateway.networking.k8s.io/v1
   kind: GRPCRoute
   metadata:
     name: example-grpcroute
     namespace: default
   spec:
     parentRefs:
       - name: eg-gateway
         namespace: envoy-gateway-system
     hostnames:
       - "grpc.example.com"
     rules:
       - matches:
           - method:
               service: "com.example.MyService"
               method: "MyMethod"
         backendRefs:
           - name: my-grpc-service
             port: 50051
   

5. TLSRoute:

   • Routes TLS traffic based on SNI (Server Name Indication) to backend services.
   • Typically used for pass-through TLS or when the backend service handles TLS termination.
   • Managed by application developers.
   yamlCopy

   apiVersion: gateway.networking.k8s.io/v1
   kind: TLSRoute
   metadata:
     name: example-tlsroute
     namespace: default
   spec:
     parentRefs:
       - name: eg-gateway
         namespace: envoy-gateway-system
     hostnames:
       - "secure.example.com"
     rules:
       - backendRefs:
           - name: my-secure-service
             port: 8443
   

Architectural Benefits

• Role-Based Separation: Clearly defines responsibilities for infrastructure, cluster, and application teams.
• Extensibility: Policies (e.g., rate limiting, authentication) can be attached at various levels (Gateway, Route, Service) using Policy CRDs.
• Protocol Awareness: Native support for HTTP, HTTPS, gRPC, and TCP/UDP, moving beyond HTTP-only limitations of Ingress.
• Portability: Standardized API reduces vendor lock-in and simplifies migration between implementations.

Introducing Envoy Gateway

Envoy Gateway is an official Envoy project designed to provision and manage Envoy proxies as a Gateway API implementation. It acts as a control plane, translating Gateway API resources into Envoy's dynamic configuration (xDS API).

Why Envoy Gateway?

• Performance: Built on Envoy Proxy, known for its high performance, low latency, and robust feature set.
• Extensibility: Leverages Envoy's filter chain architecture for advanced traffic manipulation.
• Mature Ecosystem: Benefits from the extensive features and community support of the Envoy Proxy project.
• Advanced Features: Out-of-the-box support for weighted load balancing, circuit breaking, global rate limiting, advanced observability, and more.
• Official Support: Being an official Envoy project, it ensures alignment with Envoy's roadmap and best practices.

Comparison: Ingress-NGINX vs. Envoy Gateway

| Feature / Metric | Ingress-NGINX | Envoy Gateway (with Gateway API) | Trade-offs to the next section.

## Installation and Initial Setup

This section outlines the deployment of Envoy Gateway and the initial configuration of Gateway API resources.

### Prerequisites

*   A running Kubernetes cluster (v1.24+).
*   `kubectl` configured to connect to your cluster.
*   `helm` (v3.x+) installed.

### Installing Envoy Gateway

Envoy Gateway is typically installed via Helm.

```bash
# 1. Add the Envoy Gateway Helm repository
helm repo add envoy-gateway https://envoyproxy.github.io/gateway/
helm repo update

# 2. Create a namespace for Envoy Gateway
kubectl create namespace envoy-gateway-system

# 3. Install Envoy Gateway
helm install envoy-gateway envoy-gateway/envoy-gateway -n envoy-gateway-system --version v0.0.0 # Replace with latest stable version

Verify the installation:

kubectl get pods -n envoy-gateway-system
# Expected output (example):
# NAME                                        READY   STATUS    RESTARTS   AGE
# envoy-gateway-7b8c7d4f5-abcde               1/1     Running   0          2m
# envoy-gateway-bootstrap-7b8c7d4f5-abcde     1/1     Running   0          2m
# eg-gateway-proxy-7b8c7d4f5-abcde            1/1     Running   0          2m

kubectl get gatewayclass
# Expected output:
# NAME         CONTROLLER                         ACCEPTED   AGE
# eg-standard  gateway.envoyproxy.io/gatewayclass-controller   True       2m

Deploying a Basic Gateway

After installation, the eg-standard GatewayClass is available. Now, deploy a Gateway resource that will provision an external LoadBalancer.

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-public-gateway
  namespace: envoy-gateway-system # Gateways are typically deployed in the controller's namespace
spec:
  gatewayClassName: eg-standard
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All # Allow HTTPRoutes from any namespace to attach
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: default-tls-cert # This secret will be created later with cert-manager
      allowedRoutes:
        namespaces:
          from: All # Allow HTTPRoutes from any namespace to attach

Apply the Gateway:

kubectl apply -f gateway.yaml

Monitor the Gateway status until an external IP/hostname is assigned:

kubectl get gateway eg-public-gateway -n envoy-gateway-system -w
# Expected output (example):
# NAME              CLASS         ADDRESS         READY   HTTP    HTTPS   AGE
# eg-public-gateway eg-standard   <pending>       False   80      443     10s
# ...
# eg-public-gateway eg-standard   192.0.2.123     True    80      443     2m

Note the ADDRESS assigned to your Gateway. This is the external IP/hostname you will point your DNS records to.

Core Routing Concepts with Envoy Gateway

This section demonstrates how to configure various routing patterns using Gateway API resources.

6.1. HTTPRoute

HTTPRoute is the primary resource for routing HTTP/HTTPS traffic.

Basic Path-Based Routing

Route requests to /api/v1 to my-api-service and /web to my-web-service.

# services.yaml (example backend services)
apiVersion: v1
kind: Service
metadata:
  name: my-api-service
spec:
  selector:
    app: my-api
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api-deployment
spec:
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
    spec:
      containers:
        - name: api
          image: nginxdemos/hello:latest # Placeholder image
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-web-service
spec:
  selector:
    app: my-web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-deployment
spec:
  selector:
    matchLabels:
      app: my-web
  template:
    metadata:
      labels:
        app: my-web
    spec:
      containers:
        - name: web
          image: nginxdemos/hello:latest # Placeholder image
          ports:
            - containerPort: 8080

# httproute-path.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: path-routing-example
  namespace: default
spec:
  parentRefs:
    - name: eg-public-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "app.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api/v1
      backendRefs:
        - name: my-api-service
          port: 80
    - matches:
        - path:
            type: PathPrefix
            value: /web
      backendRefs:
        - name: my-web-service
          port: 80

Apply these manifests:

kubectl apply -f services.yaml
kubectl apply -f httproute-path.yaml

Requests to http://app.example.com/api/v1/users will go to my-api-service, and http://app.example.com/web/index.html to my-web-service.

Header-Based Routing

Route traffic based on the presence or value of an HTTP header. This is useful for A/B testing or routing internal tools.

# httproute-header.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: header-routing-example
  namespace: default
spec:
  parentRefs:
    - name: eg-public-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "app.example.com"
  rules:
    - matches:
        - headers:
            - name: X-Internal-User
              value: "true"
          path:
            type: PathPrefix
            value: /admin
      backendRefs:
        - name: my-admin-service # Assume this service exists
          port: 80
    - matches:
        - path:
            type: PathPrefix
            value: /admin
      backendRefs:
        - name: my-public-service # Assume this service exists
          port: 80

In this example, requests to app.example.com/admin with X-Internal-User: true header will go to my-admin-service, otherwise to my-public-service.

6.2. GRPCRoute

GRPCRoute enables routing based on gRPC service and method names.

# grpc-service.yaml (example gRPC backend)
apiVersion: v1
kind: Service
metadata:
  name: my-grpc-service
spec:
  selector:
    app: my-grpc-app
  ports:
    - protocol: TCP
      port: 50051
      targetPort: 50051
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-grpc-deployment
spec:
  selector:
    matchLabels:
      app: my-grpc-app
  template:
    metadata:
      labels:
        app: my-grpc-app
    spec:
      containers:
        - name: grpc-server
          image: grpc/go-grpc-example-server:latest # Placeholder gRPC server
          ports:
            - containerPort: 50051

# grpcroute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: grpc-routing-example
  namespace: default
spec:
  parentRefs:
    - name: eg-public-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "grpc.example.com"
  rules:
    - matches:
        - method:
            service: "helloworld.Greeter" # gRPC service name
            method: "SayHello" # gRPC method name
            type: Exact
      backendRefs:
        - name: my-grpc-service
          port: 50051
    - matches:
        - method:
            service: "helloworld.Greeter"
            type: Exact # Matches any method under Greeter service
      backendRefs:
        - name: my-grpc-service
          port: 50051

Apply these manifests:

kubectl apply -f grpc-service.yaml
kubectl apply -f grpcroute.yaml

6.3. TLSRoute

TLSRoute is used for SNI-based routing, typically for TLS passthrough where the backend service handles TLS termination.

# tls-service.yaml (example TLS backend)
apiVersion: v1
kind: Service
metadata:
  name: my-secure-service
spec:
  selector:
    app: my-secure-app
  ports:
    - protocol: TCP
      port: 8443
      targetPort: 8443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-secure-deployment
spec:
  selector:
    matchLabels:
      app: my-secure-app
  template:
    metadata:
      labels:
        app: my-secure-app
    spec:
      containers:
        - name: secure-app
          image: nginxdemos/hello:latest # Placeholder, should be a TLS-enabled app
          ports:
            - containerPort: 8443

# tlsroute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata:
  name: tls-routing-example
  namespace: default
spec:
  parentRefs:
    - name: eg-public-gateway
      namespace: envoy-gateway-system
      sectionName: https # Attach to the HTTPS listener
  hostnames:
    - "secure.example.com"
  rules:
    - backendRefs:
        - name: my-secure-service
          port: 8443

Apply these manifests:

kubectl apply -f tls-service.yaml
kubectl apply -f tlsroute.yaml

Requests to secure.example.com on port 443 will be forwarded to my-secure-service:8443. Note that for `TLSRoute