# Kubernetes: podAffinity with NotIn vs podAntiAffinity with In

- Date: 2024-10-15
- Tags: kubernetes, mlops, devops, edge-cases, k8s
- Canonical URL: https://zablo.net/blog/post/kubernetes-pod-affinity-notin-podantiaffinity-in-operators/



At first glance, setting a `podAffinity` with the `NotIn` operator might seem equivalent to setting a `podAntiAffinity` with the `In` operator. But in **Kubernetes**, these two configurations can lead to dramatically different scheduling behaviors.

<div class="yellow-blockquote">

> 🤓 🤓 🤓 **Nerd alert**<br/>
> This blog post is about a super nerdy edge-case that I've discussed sometime ago<br/>with Wojtek from <a href="https://ml-workout.pl" target="_blank">ML-Workout.pl</a>.

</div>

## The Question

Does setting the following `podAffinity` on a `Deployment`:

```yaml
podAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: NotIn
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"
```

have the same effect as setting this `podAntiAffinity`?

```yaml
podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: In
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"
```

## The Answer

<span class="underline">No, they are not the same</span> ⬛️.

Here's why:

1. Understanding `podAffinity` with `NotIn`:

   - **What it means:** *"Schedule this pod on a node where there is at <span class="underline">least one</span> pod whose `app` label is **not** `myapp`."*
   - **Implication:** The node must have at <span class="underline">least one</span> pod with an `app` label that is **different** from `myapp`.

2. Understanding `podAntiAffinity` with `In`:

   - **What it means:** *"Avoid scheduling this pod on nodes where there is any pod whose `app` label is `myapp`."*
   - **Implication:** The pod can be scheduled on any node that doesn't have pods labeled `app: myapp`.

## Deep walk-through
Let's walk through a practical example to see how these configurations behave differently.

1. Starting Point<br/>
    You have **2 or more nodes** with **no pods** scheduled on them—just plain nodes with no taints or special configurations.

2. Deploy `app:affinity`<br/>
    You apply a Deployment labeled `app: affinity` with the following `podAffinity` rule:

    ```yaml
    podAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
            operator: NotIn
            values:
            - myapp
        topologyKey: "kubernetes.io/hostname"
    ```

3. **Result:**<br/>The pod **does not start** and stays in `Pending`.<br/>
    **Why?** Because there are **no pods** on any nodes, so there's **no node** where a pod with an `app` label **not equal to** `myapp` is running. The affinity rule requires at <span class="underline">least one</span> such pod.

4. Deploy `app:anti`:<br/>
    You deploy another Deployment labeled `app: anti` with the following `podAntiAffinity` rule:

    ```yaml
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
            operator: In
            values:
            - myapp
        topologyKey: "kubernetes.io/hostname"
    ```

5. **Result:**<br/>The pod **schedules immediately** on one of the nodes.<br/>
    **Why?** Because there are no pods with `app: myapp` running anywhere, so there's nothing to avoid.

6. Recheck `app:affinity`:<br/>
    After `app:anti` is scheduled, the `app:affinity` pod **now schedules** on the **same node** as `app:anti`.<br/>
    **Why?** Because the node now has a pod (`app: anti`) with an `app` label **not equal to** `myapp`, satisfying the affinity rule.

7. Increase the replicas of `app:affinity` to 2.

8. **Result:**<br/>The new pod **also schedules on the same node**, even if it's getting crowded 🙃
    <br/>**Why?** It's the only node that satisfies the `podAffinity` condition.

<hr/>

## DON'T BELIEVE ME, CHECK IT OUT BY YOURSELF! 👀

<hr/>

### Hands-On Example: Experimenting with Pod Affinity and Anti-Affinity

Ready to see this in action? Let's walk through a quick hands-on example using **k3d**, a lightweight Kubernetes distribution that runs inside Docker. This will help you visualize how these affinity rules affect pod scheduling.

#### Step 1: Install k3d

First, install **k3d** by running the following command:

```bash
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
# See source: https://k3d.io/v5.7.4/#installation
```

Verify the installation:

```bash
k3d version
```
#### Step 2: Create a k3d Cluster

Create a new k3d cluster with a single node:

```bash
k3d cluster create mycluster --servers 1
```

This command creates a Kubernetes cluster named `mycluster` with one server node.

#### Step 3: Set Up `kubectl`

Make sure you have `kubectl` installed and configured to interact with your new cluster:

```bash
kubectl cluster-info
```

#### Step 4: Experiment with Pod Affinity and Anti-Affinity

4.1 Apply the `affinity.yaml`

Save the following content as `affinity.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: affinity-app
  labels:
    app: affinity-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: affinity-app
  template:
    metadata:
      labels:
        app: affinity-app
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: NotIn
                values:
                - affinity-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: affinity-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1
```

Apply it:

```bash
kubectl apply -f affinity.yaml
```

4.2 Observe the Pending State

Check the status of the pods:

```bash
kubectl get pods
```

You should see that the `affinity-app` pod is in a `Pending` state.

4.3 Clean Up

Delete the deployment:

```bash
kubectl delete -f affinity.yaml
```

4.4 Apply the `anti.yaml`

Save the following content as `anti.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: anti-app
  labels:
    app: anti-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: anti-app
  template:
    metadata:
      labels:
        app: anti-app
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - anti-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: anti-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1
```

Apply it:

```bash
kubectl apply -f anti.yaml
```

4.5 Observe Pod Scheduling

Check the pods again:

```bash
kubectl get pods
```

You should see that the `anti-app` pod is running.

4.6 Clean Up and Wait

Delete the deployment:

```bash
kubectl delete -f anti.yaml
```

Wait until all pods are terminated:

```bash
kubectl get pods --watch
```

Press `Ctrl+C` to stop watching once all pods are gone.

4.7 Reapply `affinity.yaml`

Apply the `affinity.yaml` again:

```bash
kubectl apply -f affinity.yaml
```

Check the pods:

```bash
kubectl get pods
```

Again, the `affinity-app` pod should be in a `Pending` state.

4.8 Apply `anti.yaml` Again

Now, apply `anti.yaml` without deleting `affinity.yaml`:

```bash
kubectl apply -f anti.yaml
```

**Observe the Magic 🧞‍♂️🪄**

Check the pods:

```bash
kubectl get pods
```

This time, you'll see that both `affinity-app` and `anti-app` pods are running. The `affinity-app` pod could finally schedule because the node now has a pod (`anti-app`) with an `app` label not equal to `affinity-app`, satisfying the affinity condition.

#### Optional: Scale Up and Add Nodes

If you want to explore further:

- **Add More Nodes:**

  ```bash
  k3d node create extra-node --cluster mycluster
  ```

- **Increase Replicas:**

  Edit `affinity.yaml` and `anti.yaml` to increase the `replicas` count:

  ```yaml
  spec:
    replicas: 2  # Increase this number
  ```

  Reapply the deployments:

  ```bash
  kubectl apply -f affinity.yaml
  kubectl apply -f anti.yaml
  ```

Observe how the pods are scheduled across the nodes based on the affinity and anti-affinity rules.

#### Clean Up

When you're done experimenting, delete the cluster:

```bash
k3d cluster delete mycluster
```

---

By following these steps, you can see firsthand how Kubernetes handles `podAffinity` with `NotIn` and `podAntiAffinity` with `In` operators. This practical exercise should cement your understanding and help you avoid scheduling surprises in your own clusters.

---

*Enjoyed this hands-on tutorial & speaking Polish 🇵🇱? Check out more on my YouTube channel [ML-Workout](https://www.youtube.com/@ml-workout?sub_confirmation=1) for Machine Learning and MLOps tutorials!*

---


## Key Takeaways

- **Affinity Requires Presence:** `podAffinity` rules require the **presence** of pods matching the criteria.

- **Anti-Affinity Requires Absence:** `podAntiAffinity` rules require the **absence** of pods matching the criteria.

- **The `NotIn` Operator in Affinity:** When used with `podAffinity`, the `NotIn` operator looks for nodes where there are pods whose labels do **not** match the specified values, but there must be at <span class="underline">least one</span> pod present.

## Don't Let This Happen to You

Understanding these subtle differences can save you hours of debugging. Always double-check your affinity rules and remember how Kubernetes interprets them.

Please share & comment! 🗣️🗣️🗣️