Tag: sdk

The Kubernetes Operator SDK for Go

~ Davis Miller ~ Leave a comment

The Kubernetes Operator SDK for Go provides a streamlined way to build, test, and deploy Kubernetes Operators using the Go programming language. By leveraging established Kubernetes libraries like kubebuilder and controller-runtime, the Operator SDK hides much of the low-level boilerplate and lets you focus on encoding domain-specific operational logic. This detailed guide walks through the architecture, workflow, and code examples to build a Go-based Operator with the Operator SDK.

Key Concepts

  1. Operators and Controllers:
    An Operator is essentially one or more Kubernetes controllers focused on custom resource types (CRDs) that represent the desired state of a complex application. Your Operator continuously "reconciles" the actual state of the cluster and the desired state specified by the CR.
  2. Operator SDK:
    The Operator SDK provides:
    • Project scaffolding with sensible defaults and best practices.
    • Tools to create and manage CRDs, controllers, and webhooks.
    • make targets and scripts to simplify building, testing, and deploying your Operator.
    • Integration with Operator Lifecycle Manager (OLM) and OperatorHub for distribution.
  3. Go and controller-runtime:
    With Go-based Operators, you define your CustomResource (CR) types as Go structs, and implement reconciliation logic in Go. The controller-runtime library abstracts details of watches, event handling, and the reconciliation loop, making it straightforward to implement your custom logic.

Prerequisites

  • Golang: Ensure Go 1.19+ is installed.
  • kubectl: To interact with your cluster.
  • A Kubernetes Cluster: Can be local (e.g., Kind, Minikube) or remote.

Operator SDK CLI: Install from GitHub releases:

curl -LO "https://github.com/operator-framework/operator-sdk/releases/download/vX.Y.Z/operator-sdk_X.Y.Z_$(uname -m).tar.gz"
tar -xvf operator-sdk_X.Y.Z_$(uname -m).tar.gz
sudo mv operator-sdk /usr/local/bin/

Initializing a Go-Based Operator Project

Create a New Directory:

mkdir my-operator
cd my-operator

Initialize the Project:

operator-sdk init –domain=example.com –owner="MyCompany" –repo=github.com/my-org/my-operator

This command:

  • Sets up a Go module.
  • Creates directories like api/, controllers/, config/, and a Makefile.
  • Scaffolds boilerplate code including main.go.

Project Structure: After initialization, you'll see a structure similar to:

.
├─ api/
├─ controllers/
├─ config/
│  ├─ crd/
│  ├─ default/
│  ├─ manager/
│  ├─ rbac/
│  └─ samples/
├─ Makefile
├─ go.mod
└─ main.go

Defining a Custom Resource

Create a New API: Suppose you're building an Operator to manage a Memcached application. You want a custom resource Memcached with a size field.

operator-sdk create api –group=cache –version=v1alpha1 –kind=Memcached –resource –controller

Flags:

  • –resource: generates CRD and API type files.
  • –controller: generates a controller (reconciler) scaffold.

API Types: Check the newly created file api/cache/v1alpha1/memcached_types.go:

type MemcachedSpec struct {
  Size int32 `json:"size,omitempty"`
}

type MemcachedStatus struct {
  Nodes []string `json:"nodes,omitempty"`
}
  • MemcachedSpec holds the user-defined desired state.
  • MemcachedStatus reflects the observed state, updated by the Operator.

CRD Generation: Update CRDs and manifests:

make generate
make manifests

This:

  • Generates CRDs in config/crd/bases.
  • Ensures YAML manifests are up-to-date.

Install the CRD:

kubectl apply -f config/crd/bases

Now the Memcached CRD is installed into the cluster.

Implementing the Reconciliation Logic

The heart of the Operator is the controllers/memcached_controller.go file. It contains a Reconcile method that the Operator runs whenever a Memcached resource changes or related events occur (e.g., a Pod controlled by Memcached is deleted).

Typical Steps in Reconciliation:

  1. Fetch the Custom Resource:
    Get the Memcached instance from the API server using its name and namespace.
  2. Compare Desired and Actual State:
    Check how many Pods (or a Deployment replica count) currently exist. If spec.size says 3 and you have only 2 replicas, you need to add one more.
  3. Update Kubernetes Objects:
    Create, update, or delete Deployments, Services, ConfigMaps, etc., to match the desired state.
  4. Update the Status:
    After successful reconciliation, update .status fields of the Memcached resource to reflect the current state.

Example Reconciler Code:

func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Fetch the Memcached instance
    memcached := &cachev1alpha1.Memcached{}
    if err := r.Get(ctx, req.NamespacedName, memcached); err != nil {
        if errors.IsNotFound(err) {
            // Memcached resource not found, could have been deleted
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    // Define the desired Deployment
    deploy := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      memcached.Name,
            Namespace: memcached.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &memcached.Spec.Size,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{"app": "memcached"},
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{"app": "memcached"},
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "memcached",
                        Image: "memcached:1.4.36",
                        Ports: []corev1.ContainerPort{{ContainerPort: 11211}},
                    }},
                },
            },
        },
    }

    // Check if Deployment exists
    found := &appsv1.Deployment{}
    err := r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found)
    if err != nil && errors.IsNotFound(err) {
        // Create Deployment if not found
        if err := r.Create(ctx, deploy); err != nil {
            return ctrl.Result{}, err
        }
        // Newly created, requeue to verify status
        return ctrl.Result{Requeue: true}, nil
    } else if err != nil {
        return ctrl.Result{}, err
    }

    // Update Deployment if needed
    if *found.Spec.Replicas != memcached.Spec.Size {
        found.Spec.Replicas = &memcached.Spec.Size
        if err := r.Update(ctx, found); err != nil {
            return ctrl.Result{}, err
        }
    }

    // Update Memcached status (list Pods for this Deployment)
    podList := &corev1.PodList{}
    if err := r.List(ctx, podList, client.InNamespace(memcached.Namespace), client.MatchingLabels{"app": "memcached"}); err != nil {
        return ctrl.Result{}, err
    }
    podNames := []string{}
    for _, pod := range podList.Items {
        podNames = append(podNames, pod.Name)
    }

    if !equalSlices(memcached.Status.Nodes, podNames) {
        memcached.Status.Nodes = podNames
        if err := r.Status().Update(ctx, memcached); err != nil {
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

Note:
The above code is an example outline. In a real-world scenario, handle errors, check for OwnerReferences, and ensure idempotency.

Registering the Controller: The SetupWithManager function in memcached_controller.go is automatically scaffolded:

func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1alpha1.Memcached{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

This sets up watches on Memcached and owned Deployment resources.

Running and Testing the Operator

Run Locally (outside cluster):

make run

The Operator runs in your terminal and connects to the Kubernetes cluster pointed to by kubectl config current-context.

Deploy Memcached Resource: Create a Memcached instance:

apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: example-memcached
spec:
  size: 3


kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

Validate: Check Deployments:

kubectl get deployment

You should see example-memcached with 3 replicas.
Check the CR status:

kubectl get memcached example-memcached -o yaml

The .status.nodes field should list the Pod names.

Scale Up: Edit the CR and increase size to 5:

kubectl edit memcached example-memcached

The Operator will detect the change and update the Deployment to 5 replicas.

Advanced Features

Webhooks and Validation: Add admission webhooks for defaulting and validating CR fields.

operator-sdk create webhook –group=cache –version=v1alpha1 –kind=Memcached –defaulting –programmatic-validation

Implement the ValidateCreate, ValidateUpdate, Default methods in the generated files.

Versioning the CRD: Start with v1alpha1, and as your API matures, add new versions v1beta1, v1, along with conversion functions to maintain backward compatibility.

Metrics and Logging: Use built-in metrics and logging from controller-runtime to monitor Operator health and reconciliation performance.
controller-runtime automatically provides Prometheus metrics endpoints that can be scraped by Prometheus.

Testing:

  • Unit Tests: Test reconciliation logic using Go unit tests and mocks.
  • Integration Tests: Use envtest from controller-runtime to run integration tests against a fake API server.
  • Scorecard Tests: The Operator SDK includes a scorecard to test operator best practices.

Deployment with the Manager

For production scenarios, you typically run the Operator inside the cluster:

Build Image:

make docker-build IMG=quay.io/my-org/my-operator:v0.1.0

Push Image:

make docker-push IMG=quay.io/my-org/my-operator:v0.1.0

Deploy:

make deploy IMG=quay.io/my-org/my-operator:v0.1.0

This applies manifests in config/manager to create a Deployment for your Operator in the cluster.

Best Practices for Go Operators

  1. Spec and Status:
    Keep spec user-driven, status operator-driven. Update status as soon as you achieve desired state or encounter errors.
  2. Idempotent Reconciliation:
    Ensure Reconcile can run multiple times safely without causing side-effects or drifting state.
  3. Error Handling and Retries:
    Return errors to trigger automatic retries. Use backoff and requeue intervals if needed.
  4. Logging and Instrumentation:
    Add meaningful logs. Expose metrics to gauge reconciliation frequency, error counts, etc.
  5. RBAC and Security:
    Fine-tune config/rbac to grant the Operator's ServiceAccount the least privileges required.

Conclusion

Building a Go-based Operator with the Operator SDK turns what could be a complex, boilerplate-heavy process into a structured, guided workflow. You:

  • Start by initializing a project and defining APIs and CRDs.
  • Implement the reconciliation logic in Go using controller-runtime.
  • Test locally, iteratively refine, and finally deploy into a production cluster.
  • Leverage Operator SDK tools for code generation, validation, testing, and packaging.

By following these patterns and best practices, you can quickly develop powerful, maintainable Operators that fully leverage Kubernetes' extensible API model and declarative approach to complex application lifecycle management.