Kubernetes Operators

Writing a Kubernetes Operator from scratch using raw client-go requires thousands of lines of boilerplate: informers, listers, workqueues, event recorders, and reflection types.

The Operator SDK (built on top of Kubebuilder and controller-runtime) handles this scaffolding for you. It lets you focus on domain logic: defining your Custom Resource Definition (CRD) schema and implementing the reconciliation loop.

This guide walks through building a production-ready RedisCluster Operator in Go, complete with finalizers for safe cleanup, idempotent sub-resource management, and automated integration testing with envtest.

Audio Briefing• Voice: Google Cloud Journey-F

0:00 / 0:00

1. Project Initialization and Scaffolding

Install the Operator SDK and scaffold a new Go project:

bash

# Initialize operator repository
operator-sdk init \
  --domain db.example.com \
  --repo github.com/my-org/redis-operator \
  --plugins=go/v4

# Generate API (CRD) and Controller
operator-sdk create api \
  --group cache \
  --version v1alpha1 \
  --kind RedisCluster \
  --resource \
  --controller

This creates the project structure:

api/v1alpha1/rediscluster_types.go: Go struct definitions for your Custom Resource.
internal/controller/rediscluster_controller.go: The Reconcile loop.
config/: Kustomize manifests for CRDs, RBAC roles, and manager deployments.

2. Defining the CRD Schema

Edit api/v1alpha1/rediscluster_types.go to define desired state (Spec) and observed state (Status):

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// RedisClusterSpec defines the desired state of RedisCluster
type RedisClusterSpec struct {
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=9
	// +kubebuilder:default=3
	Replicas int32 `json:"replicas"`

	// +kubebuilder:validation:Required
	Version string `json:"version"`

	// +kubebuilder:default="256Mi"
	MemoryLimit string `json:"memoryLimit,omitempty"`
}

// RedisClusterStatus defines the observed state of RedisCluster
type RedisClusterStatus struct {
	Nodes      []string           `json:"nodes,omitempty"`
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Replicas",type="integer",JSONPath=".spec.replicas"
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"

// RedisCluster is the Schema for the redisclusters API
type RedisCluster struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   RedisClusterSpec   `json:"spec,omitempty"`
	Status RedisClusterStatus `json:"status,omitempty"`
}

Generate the Kubernetes CRD YAML files from the Go structs:

bash

make manifests

3. Implementing the Idempotent Reconcile Loop

The core contract of a Kubernetes controller is level-triggered reconciliation: your Reconcile function must be able to run repeatedly without causing unintended side effects.

Here is internal/controller/rediscluster_controller.go:

package controller

import (
	"context"
	"time"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/log"

	cachev1alpha1 "github.com/my-org/redis-operator/api/v1alpha1"
)

const redisFinalizer = "cache.db.example.com/finalizer"

type RedisClusterReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=cache.db.example.com,resources=redisclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cache.db.example.com,resources=redisclusters/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete

func (r *RedisClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := log.FromContext(ctx)

	// 1. Fetch the RedisCluster instance
	var cluster cachev1alpha1.RedisCluster
	if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
		if errors.IsNotFound(err) {
			return ctrl.Result{}, nil // Object deleted, ignore
		}
		return ctrl.Result{}, err
	}

	// 2. Handle Finalizers (Graceful Deletion)
	if cluster.ObjectMeta.DeletionTimestamp.IsZero() {
		// Register finalizer if not present
		if !controllerutil.ContainsFinalizer(&cluster, redisFinalizer) {
			controllerutil.AddFinalizer(&cluster, redisFinalizer)
			if err := r.Update(ctx, &cluster); err != nil {
				return ctrl.Result{}, err
			}
		}
	} else {
		// Object is being deleted
		if controllerutil.ContainsFinalizer(&cluster, redisFinalizer) {
			logger.Info("Cleaning up external Redis state...")
			// (Perform external cleanup like draining nodes or releasing cloud disks here)

			controllerutil.RemoveFinalizer(&cluster, redisFinalizer)
			if err := r.Update(ctx, &cluster); err != nil {
				return ctrl.Result{}, err
			}
		}
		return ctrl.Result{}, nil
	}

	// 3. Reconcile Underlying StatefulSet using CreateOrUpdate
	sts := &appsv1.StatefulSet{
		ObjectMeta: metav1.ObjectMeta{
			Name:      cluster.Name + "-nodes",
			Namespace: cluster.Namespace,
		},
	}

	opResult, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
		// Set owner reference for automatic garbage collection
		if err := ctrl.SetControllerReference(&cluster, sts, r.Scheme); err != nil {
			return err
		}

		labels := map[string]string{"app": "redis", "cluster": cluster.Name}
		sts.Spec.Replicas = &cluster.Spec.Replicas
		sts.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
		sts.Spec.Template = corev1.PodTemplateSpec{
			ObjectMeta: metav1.ObjectMeta{Labels: labels},
			Spec: corev1.PodSpec{
				Containers: []corev1.Container{
					{
						Name:  "redis",
						Image: "redis:" + cluster.Spec.Version,
						Resources: corev1.ResourceRequirements{
							Limits: corev1.ResourceList{
								corev1.ResourceMemory: resource.MustParse(cluster.Spec.MemoryLimit),
							},
						},
					},
				},
			},
		}
		return nil
	})

	if err != nil {
		logger.Error(err, "Failed to reconcile StatefulSet")
		return ctrl.Result{}, err
	}

	if opResult != controllerutil.OperationResultNone {
		logger.Info("StatefulSet reconciled", "operation", opResult)
	}

	// 4. Update Status
	cluster.Status.Nodes = []string{cluster.Name + "-0", cluster.Name + "-1"}
	if err := r.Status().Update(ctx, &cluster); err != nil {
		return ctrl.Result{}, err
	}

	return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

func (r *RedisClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&cachev1alpha1.RedisCluster{}).
		Owns(&appsv1.StatefulSet{}). // Re-trigger reconcile when child StatefulSet changes
		Complete(r)
}

4. Testing Controllers Fast with `envtest`

Do not deploy to a remote Kubernetes cluster for basic unit/integration tests. envtest spins up a real etcd and kube-apiserver locally in binary form in less than 2 seconds:

var _ = Describe("RedisCluster Controller", func() {
	Context("When creating a RedisCluster", func() {
		It("Should create a child StatefulSet successfully", func() {
			ctx := context.Background()
			cluster := &cachev1alpha1.RedisCluster{
				ObjectMeta: metav1.ObjectMeta{
					Name:      "test-redis",
					Namespace: "default",
				},
				Spec: cachev1alpha1.RedisClusterSpec{
					Replicas: 3,
					Version:  "7.2-alpine",
				},
			}
			Expect(k8sClient.Create(ctx, cluster)).Should(Succeed())

			// Verify StatefulSet is created by the controller
			stsLookupKey := types.NamespacedName{Name: "test-redis-nodes", Namespace: "default"}
			createdSts := &appsv1.StatefulSet{}

			Eventually(func() bool {
				err := k8sClient.Get(ctx, stsLookupKey, createdSts)
				return err == nil
			}, time.Second*10, time.Millisecond*250).Should(BeTrue())

			Expect(*createdSts.Spec.Replicas).To(Equal(int32(3)))
		})
	})
})

Run test suite:

bash

make test

5. Summary Best Practices for Kubernetes Operators

Use controllerutil.CreateOrUpdate: Prevents race conditions and overwriting external annotations.
Always register Owns() in SetupWithManager: Ensures child resource mutations trigger parent reconciliation immediately.
Use Finalizers for external state: Never delete a Custom Resource before releasing attached cloud load balancers or AWS EBS storage volumes.
Use envtest for fast CI feedback: Tests run locally without Docker or Minikube overhead.

Kubernetes Operators: Building Custom Controllers with the Operator SDK

1. Project Initialization and Scaffolding

2. Defining the CRD Schema

3. Implementing the Idempotent Reconcile Loop

4. Testing Controllers Fast with `envtest`

5. Summary Best Practices for Kubernetes Operators

You Might Also Like

Stay Updated

Free In-Browser Developer Tools