Kubernetes Advanced Deployment Strategy - Part 2

In my last post, I talked about isolating a pod to solve a complicated deployment strategy. In this post, I will walk you through isolating a pod from a Replica Set and then adding it back to its original Replica Set. Replica Sets ensure that the desired number of pods matches the current Deployment. In this post, I will be using OpenShift and its related oc CLI tool. Everything in this post applies to Kubernetes in general, and you can swap oc for kubectl.

Labels are a huge part of how Kubernetes components interact. When you create a Deployment, you may have noticed that you set different labels and label selectors. The Replica Set uses these labels to determine which Pods it's responsible for managing. For example, in the below snippet, the Replica Set will track pods with a label echoapp defined under the configuration's selector area.

...
spec:
  replicas: 3
  selector:
    matchLabels:
      app: echoapp
  template:  
    metadata:
      labels:
        app: echoapp
...

We can use this to our advantage to isolate a specific pod by modifying the labels attached to that Pod. By doing an oc edit and removing the label app: echoapp from the Pod, the Replica Set will no longer consider this Pod part of its managed Pods. Since as far as the Replica Set is concerned, there is now one fewer Pod than the Deployment specifies, so it spins up another Pod.

Let's step through an example. We'll be using this Deployment if you want to try this at home:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoapp-deployment
  labels:
    app: echoapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: echoapp
  template:  
    metadata:
      labels:
        app: echoapp
    spec:
     containers:
        - name: echo 
          image: mendhak/http-https-echo:23
          ports:
            - name: http
              protocol: TCP
              containerPort: 8080
            - name: https
              protocol: TCP
              containerPort: 8443
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"

Assuming you've saved this Deployment as deployment.yaml, you can deploy this configuration via an oc apply -f deployment.yaml.

Once you do so, running an oc get pods should give you an output similar to this:

If you edit one of the running pods with oc edit <podname>, you can remove the app: echoapp label from the Pod. Once you save the change, you should see a fourth Pod spun up by the Replication Controller.

Conversely, if we edit the Pod and add the label back to the Pod, the Replication Set will manage that Pod again. In this case, we will now have four running Pods when the Deployment calls for three. The Replication Set will now spin one of those Pods back down.

If you repeat the above steps while using the watch option of oc (oc get pods -w), you can watch the changes in real-time. I removed the label from a Pod and then added it back in the below example. You can see the Pod change states from being created, running, and finally, terminated in the image below.

One more thing of note, if you isolate a Pod using the above procedure, deleting the Deployment will not clean up the Pod. This behavior is expected since the Replication Set is not managing the lifecycle of the isolated Pod.

I hope this post gives some insight into how Replication Sets function and how easmorey it is to isolate a Pod from a Replication Set. In a future post, I will introduce you to Kubernetes Controllers / Operators and discuss how to write an Operator that can automatically isolate a Pod during a maintenance window.