Kubernetes: podAffinity with NotIn vs podAntiAffinity with In

Marcin Zabłocki — Tue, 15 Oct 2024 15:07:06 +0200

At first glance, setting a podAffinity with the NotIn operator might seem equivalent to setting a podAntiAffinity with the In operator. But in Kubernetes, these two configurations can lead to dramatically different scheduling behaviors.

🤓 🤓 🤓 Nerd alert
This blog post is about a super nerdy edge-case that I’ve discussed sometime ago
with Wojtek from ML-Workout.pl.

The Question

Does setting the following podAffinity on a Deployment:

yaml

podAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: NotIn
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

have the same effect as setting this podAntiAffinity?

yaml

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: In
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

The Answer

No, they are not the same ⬛️.

Here’s why:

Understanding podAffinity with NotIn:
- What it means: “Schedule this pod on a node where there is at least one pod whose app label is not myapp.”
- Implication: The node must have at least one pod with an app label that is different from myapp.
Understanding podAntiAffinity with In:
- What it means: “Avoid scheduling this pod on nodes where there is any pod whose app label is myapp.”
- Implication: The pod can be scheduled on any node that doesn’t have pods labeled app: myapp.

Deep walk-through

Let’s walk through a practical example to see how these configurations behave differently.

Starting Point
You have 2 or more nodes with no pods scheduled on them—just plain nodes with no taints or special configurations.

Deploy app:affinity
You apply a Deployment labeled app: affinity with the following podAffinity rule:

yaml

podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
        operator: NotIn
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

Result:
The pod does not start and stays in Pending.
Why? Because there are no pods on any nodes, so there’s no node where a pod with an app label not equal to myapp is running. The affinity rule requires at least one such pod.

Deploy app:anti:
You deploy another Deployment labeled app: anti with the following podAntiAffinity rule:

yaml

podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
        operator: In
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

Result:
The pod schedules immediately on one of the nodes.
Why? Because there are no pods with app: myapp running anywhere, so there’s nothing to avoid.
Recheck app:affinity:
After app:anti is scheduled, the app:affinity pod now schedules on the same node as app:anti.
Why? Because the node now has a pod (app: anti) with an app label not equal to myapp, satisfying the affinity rule.
Increase the replicas of app:affinity to 2.
Result:
The new pod also schedules on the same node, even if it’s getting crowded 🙃
Why? It’s the only node that satisfies the podAffinity condition.

DON’T BELIEVE ME, CHECK IT OUT BY YOURSELF! 👀

Hands-On Example: Experimenting with Pod Affinity and Anti-Affinity

Ready to see this in action? Let’s walk through a quick hands-on example using k3d, a lightweight Kubernetes distribution that runs inside Docker. This will help you visualize how these affinity rules affect pod scheduling.

Step 1: Install k3d

First, install k3d by running the following command:

bash

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
# See source: https://k3d.io/v5.7.4/#installation

Verify the installation:

bash

k3d version

Step 2: Create a k3d Cluster

Create a new k3d cluster with a single node:

bash

k3d cluster create mycluster --servers 1

This command creates a Kubernetes cluster named mycluster with one server node.

Step 3: Set Up `kubectl`

Make sure you have kubectl installed and configured to interact with your new cluster:

bash

kubectl cluster-info

Step 4: Experiment with Pod Affinity and Anti-Affinity

4.1 Apply the affinity.yaml

Save the following content as affinity.yaml:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: affinity-app
  labels:
    app: affinity-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: affinity-app
  template:
    metadata:
      labels:
        app: affinity-app
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: NotIn
                values:
                - affinity-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: affinity-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1

Apply it:

bash

kubectl apply -f affinity.yaml

4.2 Observe the Pending State

Check the status of the pods:

bash

kubectl get pods

You should see that the affinity-app pod is in a Pending state.

4.3 Clean Up

Delete the deployment:

bash

kubectl delete -f affinity.yaml

4.4 Apply the anti.yaml

Save the following content as anti.yaml:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: anti-app
  labels:
    app: anti-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: anti-app
  template:
    metadata:
      labels:
        app: anti-app
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - anti-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: anti-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1

Apply it:

bash

kubectl apply -f anti.yaml

4.5 Observe Pod Scheduling

Check the pods again:

bash

kubectl get pods

You should see that the anti-app pod is running.

4.6 Clean Up and Wait

Delete the deployment:

bash

kubectl delete -f anti.yaml

Wait until all pods are terminated:

bash

kubectl get pods --watch

Press Ctrl+C to stop watching once all pods are gone.

4.7 Reapply affinity.yaml

Apply the affinity.yaml again:

bash

kubectl apply -f affinity.yaml

Check the pods:

bash

kubectl get pods

Again, the affinity-app pod should be in a Pending state.

4.8 Apply anti.yaml Again

Now, apply anti.yaml without deleting affinity.yaml:

bash

kubectl apply -f anti.yaml

Observe the Magic 🧞‍♂️🪄

Check the pods:

bash

kubectl get pods

This time, you’ll see that both affinity-app and anti-app pods are running. The affinity-app pod could finally schedule because the node now has a pod (anti-app) with an app label not equal to affinity-app, satisfying the affinity condition.

Optional: Scale Up and Add Nodes

If you want to explore further:

Add More Nodes:

bash

k3d node create extra-node --cluster mycluster

Increase Replicas:

Edit affinity.yaml and anti.yaml to increase the replicas count:
yaml
```
spec:
  replicas: 2  # Increase this number
```
Reapply the deployments:
bash
```
kubectl apply -f affinity.yaml
kubectl apply -f anti.yaml
```

Observe how the pods are scheduled across the nodes based on the affinity and anti-affinity rules.

Clean Up

When you’re done experimenting, delete the cluster:

bash

k3d cluster delete mycluster

By following these steps, you can see firsthand how Kubernetes handles podAffinity with NotIn and podAntiAffinity with In operators. This practical exercise should cement your understanding and help you avoid scheduling surprises in your own clusters.

Enjoyed this hands-on tutorial & speaking Polish 🇵🇱? Check out more on my YouTube channel ML-Workout for Machine Learning and MLOps tutorials!

Key Takeaways

Affinity Requires Presence: podAffinity rules require the presence of pods matching the criteria.
Anti-Affinity Requires Absence: podAntiAffinity rules require the absence of pods matching the criteria.
The NotIn Operator in Affinity: When used with podAffinity, the NotIn operator looks for nodes where there are pods whose labels do not match the specified values, but there must be at least one pod present.

Don’t Let This Happen to You

Understanding these subtle differences can save you hours of debugging. Always double-check your affinity rules and remember how Kubernetes interprets them.

Please share & comment! 🗣️🗣️🗣️

edge-cases on Marcin Zabłocki blog

Kubernetes: podAffinity with NotIn vs podAntiAffinity with In

The Question

The Answer

Deep walk-through

DON’T BELIEVE ME, CHECK IT OUT BY YOURSELF! 👀

Hands-On Example: Experimenting with Pod Affinity and Anti-Affinity

Step 1: Install k3d

Step 2: Create a k3d Cluster

Step 3: Set Up kubectl

Step 4: Experiment with Pod Affinity and Anti-Affinity

Optional: Scale Up and Add Nodes

Clean Up

Key Takeaways

Don’t Let This Happen to You

Step 3: Set Up `kubectl`