Health Probes: Liveness and Readiness

Pod lifecycle phases like Running or Failed provide a high-level view of a container's state, yet they do not always indicate an application's operational status. For example, an application inside a Running container might be deadlocked, stuck in an infinite loop, or unable to connect to a database. Externally, the process may appear running, but for a user, the application is broken. Kubernetes addresses this gap with health probes, which enable the Kubelet to actively check the health of your application from within the Pod.

By configuring probes, you give Kubernetes a way to ask your application directly, "Are you okay?". Based on the answer, Kubernetes can automatically restart a broken container or temporarily stop sending it traffic, leading to more resilient and self-healing systems.

There are two primary types of probes you can configure for your containers: liveness probes and readiness probes.

Liveness Probes: Is the Application Alive?

A liveness probe checks if a container is still running and responsive. If the liveness probe fails a specified number of times, the Kubelet assumes the application is in an unrecoverable state, such as a deadlock. Its response is drastic but effective: it kills the container. Based on the Pod's restartPolicy, a new container will typically be started to replace it. This is the core mechanism for automated recovery.

Consider an application that has a memory leak. It might run fine for hours but eventually becomes unresponsive. A liveness probe can detect this unresponsiveness and trigger a restart, bringing the application back to a healthy state without manual intervention.

Here is an example of a Pod manifest with a simple HTTP liveness probe:

apiVersion: v1
kind: Pod
metadata:
  name: liveness-probe-pod
spec:
  containers:
  - name: my-app
    image: nginx
    ports:
    - containerPort: 80
    livenessProbe:
      httpGet:
        path: /
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 10

In this configuration:

• httpGet: The Kubelet will send an HTTP GET request to / on port 80 of the container. A response code between 200 and 399 is considered a success.
• initialDelaySeconds: 5: The Kubelet will wait 5 seconds after the container starts before performing the first probe. This gives the application time to initialize.
• periodSeconds: 10: The probe will be executed every 10 seconds.

Readiness Probes: Is the Application Ready to Serve Traffic?

A readiness probe determines if a container is ready to start accepting network traffic. A Pod can be alive but not yet ready. For example, an application might need to load a large configuration file or warm up a cache on startup. During this time, it's running (and would pass a liveness probe), but sending it user traffic would result in errors.

If a readiness probe fails, the Kubelet does not restart the container. Instead, it signals to the Kubernetes control plane that this Pod is not ready. The Endpoints controller then removes the Pod's IP address from any Services that are pointing to it. Traffic will not be routed to the Pod until its readiness probe starts succeeding again. This is essential for achieving zero-downtime deployments and updates.

The decision flow for liveness and readiness probes. A liveness failure leads to a container restart, while a readiness failure leads to traffic being rerouted away from the Pod.

Probe Mechanisms

Kubernetes offers three ways to implement a probe check:

1. httpGet: Performs an HTTP GET request against a specific path and port on the container's IP address. This is the most common type of probe for web applications.

   livenessProbe:
     httpGet:
       path: /healthz
       port: 8080
   

2. tcpSocket: Attempts to open a TCP socket on a specified port. If the connection can be established, the probe is successful. This is useful for applications that are not HTTP-based, such as a database.

   readinessProbe:
     tcpSocket:
       port: 5432
   

3. exec: Executes a command inside the container. The probe is successful if the command exits with a status code of 0. This gives you maximum flexibility to write a custom health check script.

   livenessProbe:
     exec:
       command:
       - cat
       - /tmp/health
   

   This exec probe succeeds only if the file /tmp/health exists and is readable inside the container.

Configuring Probe Behavior

Several parameters allow you to fine-tune probe timing to match your application's behavior. Misconfiguring these can lead to containers being restarted unnecessarily or traffic being sent to unready applications.

• initialDelaySeconds: The number of seconds to wait after the container has started before the first probe is initiated. It is important to set this long enough to allow your application to start.
• periodSeconds: How often (in seconds) to perform the probe. The default is 10.
• timeoutSeconds: The number of seconds after which the probe times out. The default is 1. If a probe times out, it is considered a failure.
• failureThreshold: After a probe fails, Kubernetes will try failureThreshold times before marking the container as unhealthy (for liveness) or not ready (for readiness). The default is 3.
• successThreshold: The minimum number of consecutive successes required for the probe to be considered successful after it has failed. The default is 1.

Here is a more complete example showing both a liveness and a readiness probe with detailed configuration:

apiVersion: v1
kind: Pod
metadata:
  name: fully-probed-pod
spec:
  containers:
  - name: api-server
    image: my-company/api-server:1.2.0
    ports:
    - containerPort: 8080

    # Check if the app is alive; if not, restart it
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      timeoutSeconds: 2
      periodSeconds: 20
      failureThreshold: 3

    # Check if the app is ready to serve traffic
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      timeoutSeconds: 1
      periodSeconds: 10
      failureThreshold: 2

In this manifest, the readiness probe starts checking sooner (initialDelaySeconds: 5) and more frequently (periodSeconds: 10) than the liveness probe. This allows Kubernetes to quickly determine if the application is ready for traffic. The liveness probe is more patient (initialDelaySeconds: 15), giving the application more time to recover from temporary issues before deciding to restart it. By using distinct /healthz and /readyz endpoints, the application can independently report its liveness and readiness status.

Properly configured health probes are fundamental to building reliable applications on Kubernetes. They transform your deployment from a simple process manager into a self-healing system that can automatically handle common application failures.