Kubernetes: podAffinity with NotIn vs podAntiAffinity with In

At first glance, setting a podAffinity with the NotIn operator might seem equivalent to setting a podAntiAffinity with the In operator. But in Kubernetes, these two configurations can lead to dramatically different scheduling behaviors.

The Question

Does setting the following podAffinity on a Deployment:

podAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: NotIn
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

have the same effect as setting this podAntiAffinity?

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: In
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

The Answer

No, they are not the same ⬛️.

Here’s why:

1. Understanding podAffinity with NotIn:

   • What it means: “Schedule this pod on a node where there is at least one pod whose app label is not myapp.”
   • Implication: The node must have at least one pod with an app label that is different from myapp.

2. Understanding podAntiAffinity with In:

   • What it means: “Avoid scheduling this pod on nodes where there is any pod whose app label is myapp.”
   • Implication: The pod can be scheduled on any node that doesn’t have pods labeled app: myapp.

Deep walk-through

Let’s walk through a practical example to see how these configurations behave differently.

1. Starting Point
   You have 2 or more nodes with no pods scheduled on them—just plain nodes with no taints or special configurations.

2. Deploy app:affinity
   You apply a Deployment labeled app: affinity with the following podAffinity rule:

   podAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
       - labelSelector:
           matchExpressions:
           - key: app
           operator: NotIn
           values:
           - myapp
       topologyKey: "kubernetes.io/hostname"

3. Result:
   The pod does not start and stays in Pending.
   Why? Because there are no pods on any nodes, so there’s no node where a pod with an app label not equal to myapp is running. The affinity rule requires at least one such pod.

4. Deploy app:anti:
   You deploy another Deployment labeled app: anti with the following podAntiAffinity rule:

   podAntiAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
       - labelSelector:
           matchExpressions:
           - key: app
           operator: In
           values:
           - myapp
       topologyKey: "kubernetes.io/hostname"

5. Result:
   The pod schedules immediately on one of the nodes.
   Why? Because there are no pods with app: myapp running anywhere, so there’s nothing to avoid.

6. Recheck app:affinity:
   After app:anti is scheduled, the app:affinity pod now schedules on the same node as app:anti.
   Why? Because the node now has a pod (app: anti) with an app label not equal to myapp, satisfying the affinity rule.

7. Increase the replicas of app:affinity to 2.

8. Result:
   The new pod also schedules on the same node, even if it’s getting crowded 🙃
   Why? It’s the only node that satisfies the podAffinity condition.

DON’T BELIEVE ME, CHECK IT OUT BY YOURSELF! 👀

Hands-On Example: Experimenting with Pod Affinity and Anti-Affinity

Ready to see this in action? Let’s walk through a quick hands-on example using k3d, a lightweight Kubernetes distribution that runs inside Docker. This will help you visualize how these affinity rules affect pod scheduling.

Step 1: Install k3d

First, install k3d by running the following command:

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
# See source: https://k3d.io/v5.7.4/#installation

Verify the installation:

k3d version

Step 2: Create a k3d Cluster

Create a new k3d cluster with a single node:

k3d cluster create mycluster --servers 1

This command creates a Kubernetes cluster named mycluster with one server node.

Step 3: Set Up kubectl

Make sure you have kubectl installed and configured to interact with your new cluster:

kubectl cluster-info

Step 4: Experiment with Pod Affinity and Anti-Affinity

4.1 Apply the affinity.yaml

Save the following content as affinity.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: affinity-app
  labels:
    app: affinity-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: affinity-app
  template:
    metadata:
      labels:
        app: affinity-app
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: NotIn
                values:
                - affinity-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: affinity-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1

Apply it:

kubectl apply -f affinity.yaml

4.2 Observe the Pending State

Check the status of the pods:

kubectl get pods

You should see that the affinity-app pod is in a Pending state.

4.3 Clean Up

Delete the deployment:

kubectl delete -f affinity.yaml

4.4 Apply the anti.yaml

Save the following content as anti.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: anti-app
  labels:
    app: anti-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: anti-app
  template:
    metadata:
      labels:
        app: anti-app
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - anti-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: anti-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1

Apply it:

kubectl apply -f anti.yaml

4.5 Observe Pod Scheduling

Check the pods again:

kubectl get pods

You should see that the anti-app pod is running.

4.6 Clean Up and Wait

Delete the deployment:

kubectl delete -f anti.yaml

Wait until all pods are terminated:

kubectl get pods --watch

Press Ctrl+C to stop watching once all pods are gone.

4.7 Reapply affinity.yaml

Apply the affinity.yaml again:

kubectl apply -f affinity.yaml

Check the pods:

kubectl get pods

Again, the affinity-app pod should be in a Pending state.

4.8 Apply anti.yaml Again

Now, apply anti.yaml without deleting affinity.yaml:

kubectl apply -f anti.yaml

Observe the Magic 🧞‍♂️🪄

Check the pods:

kubectl get pods

This time, you’ll see that both affinity-app and anti-app pods are running. The affinity-app pod could finally schedule because the node now has a pod (anti-app) with an app label not equal to affinity-app, satisfying the affinity condition.

Optional: Scale Up and Add Nodes

If you want to explore further:

• Add More Nodes:

  k3d node create extra-node --cluster mycluster

• Increase Replicas:

  Edit affinity.yaml and anti.yaml to increase the replicas count:

  spec:
    replicas: 2  # Increase this number

  Reapply the deployments:

  kubectl apply -f affinity.yaml
  kubectl apply -f anti.yaml

Observe how the pods are scheduled across the nodes based on the affinity and anti-affinity rules.

Clean Up

When you’re done experimenting, delete the cluster:

k3d cluster delete mycluster

By following these steps, you can see firsthand how Kubernetes handles podAffinity with NotIn and podAntiAffinity with In operators. This practical exercise should cement your understanding and help you avoid scheduling surprises in your own clusters.

Enjoyed this hands-on tutorial & speaking Polish 🇵🇱? Check out more on my YouTube channel ML-Workout for Machine Learning and MLOps tutorials!

Key Takeaways

• Affinity Requires Presence: podAffinity rules require the presence of pods matching the criteria.

• Anti-Affinity Requires Absence: podAntiAffinity rules require the absence of pods matching the criteria.

• The NotIn Operator in Affinity: When used with podAffinity, the NotIn operator looks for nodes where there are pods whose labels do not match the specified values, but there must be at least one pod present.

Don’t Let This Happen to You

Understanding these subtle differences can save you hours of debugging. Always double-check your affinity rules and remember how Kubernetes interprets them.

Please share & comment! 🗣️🗣️🗣️