Demystifying the Kubernetes Operator Pattern: Custom Resources, Informers, and Reconcile Loops

Table of contents

Why I Started Digging Into Operators

Early in my journey with Kubernetes, I thought deploying applications was simple: wrap your microservice in a Deployment, expose it with a Service, write a Helm chart, and let Kubelet do the magic.

For stateless HTTP microservices, that mental model works brilliantly. If a node dies, Kubernetes spins up a new pod on another node, updates the Service routing, and nobody notices.

However, when you start running stateful, distributed software—like PostgreSQL clusters, Apache Kafka, Redis Enterprise, or workflow engines like Camunda—that simple model falls apart:

• If a primary database node crashes, you can’t just spin up a blank container. You need to promote a read-replica to primary, reconfigure connection strings, initiate WAL streaming replay, and update application credentials.
• If a Kafka broker node restarts, you must ensure its in-sync replicas (ISR) catch up before reassigning partition leadership to avoid data loss.
• Performing a database backup requires coordinating application quiescence, taking a consistent snapshot, and archiving WAL logs to Object Storage (S3).

Historically, human SREs and DBAs handled these tasks with complex manual runbooks. The Kubernetes Operator Pattern automates this by embedding human operational domain knowledge directly into software controllers running inside the cluster.

1. What Is the Operator Pattern?

Coined by CoreOS in 2016 and codified in the official Kubernetes Operator documentation, the Operator pattern extends Kubernetes to manage custom, complex applications automatically.

At its core, the formula is:

$$\text{Operator} = \text{Custom Resource Definition (CRD)} + \text{Custom Controller (Reconcile Loop)}$$

The Three Pillars of an Operator

1. Custom Resource Definition (CRD): Defines new API endpoints in Kubernetes (e.g. kind: Kafka or kind: PostgresCluster). It specifies the desired configuration schema (spec) and current health status (status).
2. Custom Controller: An active daemon running inside a Pod that watches for changes to these Custom Resources and executes reconciliation logic.
3. Level-Triggered Architecture: Unlike edge-triggered systems that react to transient state changes (e.g. “a pod just died”), Kubernetes is level-triggered. The controller continuously focuses on current actual state vs. desired state. Even if 10 network packets are dropped, the next reconcile iteration will bring the cluster back to the desired spec.

2. Helm vs. StatefulSet vs. Operator: The Mental Model

A common question engineers ask when introducing Operators is: “We already use Helm and StatefulSets. Why do we need Operators?”

Here is how I differentiate their responsibilities:

3. Under the Hood: Informers, WorkQueues, and Watch Events

Operators don’t poll the Kubernetes API server every few seconds with GET requests (which would overload etcd and the API server). Instead, they leverage the Informer pattern built into client-go / controller-runtime (or Fabric8 in Java):

1. Watch Streams: The controller opens a long-lived HTTP Watch connection to the API Server using HTTP/2 chunked streaming or WebSockets.
2. Reflector & Delta FIFO Queue: When a resource is created, updated, or deleted, the API Server pushes lightweight event notifications containing a ResourceVersion. The Reflector puts these keys into a Delta FIFO queue.
3. Local Store (Cache): The Informer updates an in-memory cache (Indexer), so reads inside Reconcile never touch the API Server directly.
4. WorkQueue: Events are pushed onto a rate-limiting WorkQueue, which worker goroutines pick up to execute Reconcile(ctx, req).

4. Deep-Dive: Anatomy of a Reconcile Function

In Go (using controller-runtime or operator-sdk), the controller’s heart is the Reconcile method.

Here is a detailed, realistic example of how a custom cluster operator reconciles state:

package controllers

import (
	"context"
	"time"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"

	databasev1 "github.com/example/postgres-operator/api/v1"
)

type PostgresClusterReconciler struct {
	client.Client
}

func (r *PostgresClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// 1. OBSERVE: Fetch the latest Custom Resource from API Server (via local cache)
	var cluster databasev1.PostgresCluster
	if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
		if errors.IsNotFound(err) {
			// Object deleted; garbage collection / finalizers handle cleanup
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	// 2. COMPARE: Ensure underlying StatefulSet exists and matches Spec
	var statefulSet appsv1.StatefulSet
	err := r.Get(ctx, req.NamespacedName, &statefulSet)
	if errors.IsNotFound(err) {
		// 3. ACT: Define and create the StatefulSet for PostgreSQL
		newSts := r.buildStatefulSet(&cluster)
		if err := r.Create(ctx, newSts); err != nil {
			return ctrl.Result{}, err
		}
		return ctrl.Result{Requeue: true}, nil
	}

	// 4. CHECK FAILOVER: Query pod health to check if Primary is responding
	primaryHealthy := r.checkPrimaryHealth(ctx, &cluster)
	if !primaryHealthy {
		// Perform active failover: promote replica to primary
		if err := r.promoteReplicaToPrimary(ctx, &cluster); err != nil {
			return ctrl.Result{RequeueAfter: 5 * time.Second}, err
		}
	}

	// 5. UPDATE STATUS: Update CR status subresource with active metrics
	cluster.Status.ReadyReplicas = statefulSet.Status.ReadyReplicas
	cluster.Status.Phase = "Running"
	_ = r.Status().Update(ctx, &cluster)

	// Requeue periodically for periodic health checks
	return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

Critical Concept: Idempotency & Finalizers

• Idempotency: Notice how Reconcile is structured: it doesn’t assume it was called once. If network packets drop or the controller restarts mid-operation, Reconcile will run again. Executing Reconcile 1 time or 100 times sequentially against the same cluster state yields the exact same side effects without duplicating pods or corrupting data.
• Finalizers: When a Custom Resource is deleted (kubectl delete), Kubernetes sets deletionTimestamp. If a finalizer string is registered in metadata.finalizers, Kubernetes delays hard deletion until the Operator cleans up external resources (e.g. detaching cloud storage, closing DB connections) and removes the finalizer key.

5. Bridging to Strimzi: How Kafka Uses the Operator Pattern

One of the best real-world examples of the Operator pattern is Strimzi (the CNCF Apache Kafka Operator for Kubernetes).

Instead of running manual kafka-topics.sh scripts or SSHing into brokers, Strimzi translates Kafka cluster primitives into Kubernetes Custom Resources:

A. Kafka CRD (Cluster Operator)

Manages the lifecycle of Kafka brokers, KRaft / Zookeeper nodes, JVM memory settings, JBOD storage mounts, and TLS certificates:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: production-kafka
spec:
  kafka:
    version: 3.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 100Gi
          deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

B. KafkaTopic CRD (Topic Operator)

Defines a Kafka topic declaratively. When you apply this manifest, the Strimzi Topic Operator’s reconcile loop invokes the Kafka Admin Client API to create or alter the topic inside the live brokers:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: order-events
  labels:
    strimzi.io/cluster: production-kafka
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: 604800000 # 7 days
    segment.bytes: 1073741824 # 1 GB

C. KafkaUser CRD (User Operator)

Manages SASL / mTLS client authentication credentials and ACL authorization rules declaratively. Strimzi generates the K8s Secret containing client certificates and updates Kafka ACLs automatically.

6. Famous Production Operators in the Wild

1. Strimzi Kafka Operator: Declarative management of Kafka brokers (Kafka), topics (KafkaTopic), users (KafkaUser), and Kafka Connect plugins.
2. CloudNativePG (CNPG): Handles PostgreSQL physical streaming replication, automatic failover with zero data loss, continuous WAL archiving to S3/MinIO, and point-in-time recovery (PITR).
3. Prometheus Operator: Introduces ServiceMonitor and PrometheusRule CRDs. Applying a ServiceMonitor alongside your microservice causes Prometheus to dynamically discover and scrape metrics.
4. ngrok Operator: Replaces legacy ingress controllers by automatically mapping Kubernetes Services to secure public ngrok edge tunnels using custom HTTPOption and Domain CRDs.

Summary & Key Takeaways

• Declarative Operations: The Operator pattern bridges user intent (spec) with automated execution loops (status).
• Informers Keep It Fast: Efficient controllers use Informers, HTTP Watch streams, and local caches rather than polling etcd.
• Level-Triggered & Idempotent: Always design Reconcile logic to evaluate current actual state vs desired spec safely across repeated iterations.
• Foundational for Strimzi: Understanding CRDs and Reconcile Loops makes managing Kafka with Strimzi natural and intuitive.