Marcin Zabłocki blog

How to connect to Home Assistant from anywhere using NordVPN meshnet

Marcin Zabłocki — Sat, 03 Jan 2026 15:52:00 +0000

So you have your Home Assistant working just fine, but you can only access it on your mobile app while you’re at home. I had the same problem, but I wanted to be able to play with my LED lights and power switches remotely, without having to pay for Home Assistant Cloud. I already had NordVPN at hand and I knew it had Meshnet, so it was a great opportunity to use it.

What is NordVPN Meshnet?

Meshnet is NordVPN’s peer-to-peer networking feature that lets you connect multiple devices into a private virtual network. Unlike traditional VPN services that route traffic through a central server, Meshnet creates direct connections between your devices, giving you lower latency and the ability to access services (like Home Assistant) on any device in your mesh without exposing them to the public internet.

The best part? Meshnet is completely free. You don’t need a special subscription beyond the standard NordVPN subscription, and it works on Windows, macOS, Linux, iOS, and Android. You can add up to 60 devices to a single Meshnet, which is more than enough for most home setups.

Each device in your Meshnet gets assigned a unique device name (like homeassistant-linux in my case, which is the actual NordVPN Meshnet name of my Home Assistant server) and a private IP address in the 100.64.0.0/10 range. This makes it easy to remember and access your services by name, similar to how you’d use DNS.

How it works in practice

When you enable Meshnet on both your Home Assistant server and your client device, they can communicate directly over an encrypted tunnel. Port forwarding, dynamic DNS, and exposing your server to the internet are all unnecessary. It’s just a private network between your devices.

The problem I ran into (and you probably will too) is that Meshnet blocks access to Docker containers and local networks by default. This guide shows you exactly how to fix it.

Quick solution summary

If you just want the steps without the explanation, here’s what you need to do:

Enable UFW firewall on your Linux server
Add a rule: sudo ufw allow in on nordlynx to any port 8123 proto tcp
Grant Meshnet access: sudo nordvpn meshnet peer local allow <your-device-name>
Test with: curl http://homeassistant-linux:8123 (replace with your actual Meshnet device name)

The key insight: Meshnet treats Docker networks as “local” and blocks access by default. That’s why SSH works but Home Assistant doesn’t. The peer local allow command fixes this.

Meshnet configuration on iOS

Once everything is set up correctly, you will be able to access Home Assistant remotely, e.g. when you’re out of home on iOS:

Prerequisites

Before you start, make sure you have:

A Linux server running Home Assistant (preferably in Docker with docker-compose)
NordVPN installed on that Linux server
NordVPN Meshnet enabled on both your Home Assistant server and your client device
UFW firewall (Uncomplicated Firewall) installed on the Linux server
At least one other device with NordVPN Meshnet enabled to test the connection

If you’re running Home Assistant in Docker, your setup should look something like this:

yaml

# ... other services skipped
services:
  homeassistant:
    container_name: homeassistant
    image: ghcr.io/home-assistant/home-assistant:stable
    restart: always
    volumes:
      - ./data/homeassistant:/config
    ports:
      - 8123:8123
    networks:
      - ha_net

networks:
  ha_net:
    driver: bridge

Diagnostic steps: How to debug “Home Assistant not working on meshnet”

Check if Home Assistant is actually listening

From the Linux server itself, verify the port is exposed:

bash

docker ps | grep homeassistant

You should see 0.0.0.0:8123->8123/tcp in the PORTS column. If you see something else, your Docker port mapping is misconfigured.

Also check if something is listening:

bash

sudo ss -tulpn | grep 8123

bash

sudo netstat -tulpn | grep 8123

You should see a line with LISTEN on 0.0.0.0:8123 or :::8123.

Verify the local network path works first

From another device on your LAN, try accessing Home Assistant using the server’s local IP:

bash

curl -v http://192.168.x.x:8123

or if you prefer a simple port check:

bash

nc -vz 192.168.x.x 8123

You should get an HTTP response or “succeeded” message. If this fails, your Docker setup or local network is broken. Fix this before moving on.

Test the meshnet connection itself

From your client device that has Meshnet enabled, try to reach SSH first:

bash

nc -vz <meshnet-device-name> 22

Replace <meshnet-device-name> with the device name from Meshnet (like the homeassistant-linux name in your Meshnet settings). If SSH works, Meshnet itself is working.

Now try Home Assistant:

bash

nc -vz <meshnet-device-name> 8123

If SSH succeeds but this fails, you’ve found the exact problem: Meshnet is blocking traffic to port 8123 specifically.

Understand why SSH works but Home Assistant doesn’t

SSH works because it binds to the host’s nordlynx interface directly. Home Assistant runs inside a Docker container on a bridge network (172.17.0.0/16 range by default). Meshnet treats Docker networks as “local networks” and blocks peer access to them unless you explicitly grant permission.

The complete solution: step-by-step

Step 1: enable UFW

First, enable UFW on your Linux server (if not already enabled):

bash

sudo ufw enable

Check current status:

bash

sudo ufw status

Step 2: allow meshnet traffic on port 8123

The key command that makes Home Assistant accessible over Meshnet:

bash

sudo ufw allow in on nordlynx to any port 8123 proto tcp

This tells the firewall: “If traffic comes in via the NordVPN Meshnet interface (nordlynx) on TCP port 8123, allow it.”

Step 3: reload UFW

Apply the changes immediately:

bash

sudo ufw reload

Verify the rule was added:

bash

sudo ufw status numbered

You should see something like:

text

[ N] 8123/tcp on nordlynx  ALLOW       Anywhere

Step 4: grant meshnet local network access

This is the critical step that most guides miss. On your Linux server, run:

bash

nordvpn meshnet peer list

This will show all connected Meshnet peers. Find the device you’re connecting from (like the Meshnet device name shown in your NordVPN app).

Then allow that device to access local networks:

bash

nordvpn meshnet peer local allow <your-connecting-device-name>

For example:

bash

nordvpn meshnet peer local allow secret.cat-everest.nord

This is essential. Docker containers live on a bridge network, which Meshnet classifies as a “local network.” By default, Meshnet blocks peer access to local networks. This command explicitly grants permission.

If you want to allow all Meshnet peers (less secure, but simpler), you can run:

bash

nordvpn meshnet peer local allow all

Step 5: test the connection

From your client device, try accessing Home Assistant using the Meshnet device name:

bash

curl http://homeassistant-linux:8123

Or open a browser and navigate to:

text

http://homeassistant-linux:8123

Replace homeassistant-linux with whatever your actual Meshnet device name is.

It should work now. If it doesn’t, double-check that:

The device name in meshnet peer local allow matches exactly (case-sensitive)
UFW is allowing traffic on the nordlynx interface
Home Assistant is still running and listening on 8123

That’s all

Good luck and happy Zigbee’ing 🐝

Kubernetes: podAffinity with NotIn vs podAntiAffinity with In

Marcin Zabłocki — Tue, 15 Oct 2024 15:07:06 +0200

At first glance, setting a podAffinity with the NotIn operator might seem equivalent to setting a podAntiAffinity with the In operator. But in Kubernetes, these two configurations can lead to dramatically different scheduling behaviors.

🤓 🤓 🤓 Nerd alert
This blog post is about a super nerdy edge-case that I’ve discussed sometime ago
with Wojtek from ML-Workout.pl.

The Question

Does setting the following podAffinity on a Deployment:

yaml

podAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: NotIn
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

have the same effect as setting this podAntiAffinity?

yaml

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
  - labelSelector:
      matchExpressions:
      - key: app
        operator: In
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

The Answer

No, they are not the same ⬛️.

Here’s why:

Understanding podAffinity with NotIn:
- What it means: “Schedule this pod on a node where there is at least one pod whose app label is not myapp.”
- Implication: The node must have at least one pod with an app label that is different from myapp.
Understanding podAntiAffinity with In:
- What it means: “Avoid scheduling this pod on nodes where there is any pod whose app label is myapp.”
- Implication: The pod can be scheduled on any node that doesn’t have pods labeled app: myapp.

Deep walk-through

Let’s walk through a practical example to see how these configurations behave differently.

Starting Point
You have 2 or more nodes with no pods scheduled on them—just plain nodes with no taints or special configurations.

Deploy app:affinity
You apply a Deployment labeled app: affinity with the following podAffinity rule:

yaml

podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
        operator: NotIn
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

Result:
The pod does not start and stays in Pending.
Why? Because there are no pods on any nodes, so there’s no node where a pod with an app label not equal to myapp is running. The affinity rule requires at least one such pod.

Deploy app:anti:
You deploy another Deployment labeled app: anti with the following podAntiAffinity rule:

yaml

podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
        operator: In
        values:
        - myapp
    topologyKey: "kubernetes.io/hostname"

Result:
The pod schedules immediately on one of the nodes.
Why? Because there are no pods with app: myapp running anywhere, so there’s nothing to avoid.
Recheck app:affinity:
After app:anti is scheduled, the app:affinity pod now schedules on the same node as app:anti.
Why? Because the node now has a pod (app: anti) with an app label not equal to myapp, satisfying the affinity rule.
Increase the replicas of app:affinity to 2.
Result:
The new pod also schedules on the same node, even if it’s getting crowded 🙃
Why? It’s the only node that satisfies the podAffinity condition.

DON’T BELIEVE ME, CHECK IT OUT BY YOURSELF! 👀

Hands-On Example: Experimenting with Pod Affinity and Anti-Affinity

Ready to see this in action? Let’s walk through a quick hands-on example using k3d, a lightweight Kubernetes distribution that runs inside Docker. This will help you visualize how these affinity rules affect pod scheduling.

Step 1: Install k3d

First, install k3d by running the following command:

bash

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
# See source: https://k3d.io/v5.7.4/#installation

Verify the installation:

bash

k3d version

Step 2: Create a k3d Cluster

Create a new k3d cluster with a single node:

bash

k3d cluster create mycluster --servers 1

This command creates a Kubernetes cluster named mycluster with one server node.

Step 3: Set Up `kubectl`

Make sure you have kubectl installed and configured to interact with your new cluster:

bash

kubectl cluster-info

Step 4: Experiment with Pod Affinity and Anti-Affinity

4.1 Apply the affinity.yaml

Save the following content as affinity.yaml:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: affinity-app
  labels:
    app: affinity-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: affinity-app
  template:
    metadata:
      labels:
        app: affinity-app
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: NotIn
                values:
                - affinity-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: affinity-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1

Apply it:

bash

kubectl apply -f affinity.yaml

4.2 Observe the Pending State

Check the status of the pods:

bash

kubectl get pods

You should see that the affinity-app pod is in a Pending state.

4.3 Clean Up

Delete the deployment:

bash

kubectl delete -f affinity.yaml

4.4 Apply the anti.yaml

Save the following content as anti.yaml:

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: anti-app
  labels:
    app: anti-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: anti-app
  template:
    metadata:
      labels:
        app: anti-app
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - anti-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: anti-app
        image: alpine:3.14
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo 'Running' $(date); sleep 5; done"]
      terminationGracePeriodSeconds: 1

Apply it:

bash

kubectl apply -f anti.yaml

4.5 Observe Pod Scheduling

Check the pods again:

bash

kubectl get pods

You should see that the anti-app pod is running.

4.6 Clean Up and Wait

Delete the deployment:

bash

kubectl delete -f anti.yaml

Wait until all pods are terminated:

bash

kubectl get pods --watch

Press Ctrl+C to stop watching once all pods are gone.

4.7 Reapply affinity.yaml

Apply the affinity.yaml again:

bash

kubectl apply -f affinity.yaml

Check the pods:

bash

kubectl get pods

Again, the affinity-app pod should be in a Pending state.

4.8 Apply anti.yaml Again

Now, apply anti.yaml without deleting affinity.yaml:

bash

kubectl apply -f anti.yaml

Observe the Magic 🧞‍♂️🪄

Check the pods:

bash

kubectl get pods

This time, you’ll see that both affinity-app and anti-app pods are running. The affinity-app pod could finally schedule because the node now has a pod (anti-app) with an app label not equal to affinity-app, satisfying the affinity condition.

Optional: Scale Up and Add Nodes

If you want to explore further:

Add More Nodes:

bash

k3d node create extra-node --cluster mycluster

Increase Replicas:

Edit affinity.yaml and anti.yaml to increase the replicas count:
yaml
```
spec:
  replicas: 2  # Increase this number
```
Reapply the deployments:
bash
```
kubectl apply -f affinity.yaml
kubectl apply -f anti.yaml
```

Observe how the pods are scheduled across the nodes based on the affinity and anti-affinity rules.

Clean Up

When you’re done experimenting, delete the cluster:

bash

k3d cluster delete mycluster

By following these steps, you can see firsthand how Kubernetes handles podAffinity with NotIn and podAntiAffinity with In operators. This practical exercise should cement your understanding and help you avoid scheduling surprises in your own clusters.

Enjoyed this hands-on tutorial & speaking Polish 🇵🇱? Check out more on my YouTube channel ML-Workout for Machine Learning and MLOps tutorials!

Key Takeaways

Affinity Requires Presence: podAffinity rules require the presence of pods matching the criteria.
Anti-Affinity Requires Absence: podAntiAffinity rules require the absence of pods matching the criteria.
The NotIn Operator in Affinity: When used with podAffinity, the NotIn operator looks for nodes where there are pods whose labels do not match the specified values, but there must be at least one pod present.

Don’t Let This Happen to You

Understanding these subtle differences can save you hours of debugging. Always double-check your affinity rules and remember how Kubernetes interprets them.

Please share & comment! 🗣️🗣️🗣️

Twitter Sentiment Analysis on BigQuery using ONNX + Huggingface Transformers

Marcin Zabłocki — Sun, 14 May 2023 01:00:00 +0000

This is my second blog post about running machine learning / deep learning models in BigQuery. This time, I will use the latest (still in preview) capabilities of BigQuery ML that allow to run ONNX models within the BigQuery itself.

In this blogpost dig into this new feature, to explore it’s capabilities and limitations (there are few!). As an example, I will deploy a DistilBERT-based classifier to analyse the sentiment of tweets stored in a BigQuery table.

If you’re interested in deploying custom models to BigQuery (that cannot be converted to ONNX), check out my previous blogpost: Deploy MLflow Models On BigQuery.

TL;DR

You will learn how to deploy ONNX models to BigQuery ML & understand the limitations.

What is not in the scope of this blogpost

For brevity, I’m not training any models here - I’m reusing a publicly available model.

What you will find below

Prerequisites

You will need:

Python (I’m using 3.10)
access to BigQuery
access to Cloud Functions
access to Google Cloud Storage bucket
Libraries: transformers, datasets, pandas, pandas-gbq, onnxruntime, onnx

Dataset

To make this blogpost both self-contained and complete, here’s how you can easily upload a portion of tweet_eval dataset into BigQuery. If you have some text already in BigQuery, you can skip this part.

python

import pandas as pd
from datasets import load_dataset

dataset = load_dataset('tweet_eval', 'sentiment')
validation_subset = dataset['validation'] # <-- use any split you want, here it's just for the demo purposes

df = pd.DataFrame(validation_subset)
df.to_gbq('onnx_demo.twitter_sentiment_validation', '<gcp-project-id>', if_exists='replace')

Preparing the model

The task is simple - find or train a model and convert in to ONNX format. I’m using the existing one: distilbert-base-uncased-finetuned-sst-2-english - it’s trained on 2 classes (positive and negative), not on tweets, but for the tutorial purposes that’s irrelevant - the goal is to deploy it and run the inference, so the steps for any other model will be the same.

⚠️ Limitation #1
BigQuery ML inference engine only supports ONNX models that are smaller than 450 MB. This is a serious limitation - to put this into context - deploying a popular sentiment-analysis model cardiffnlp/twitter-roberta-base-sentiment based on RoBERTa is NOT possible, because after the conversion to ONNX format, it’s size is 476MB.
State for 2023-05-14

To convert the model into ONNX format, you can just run the module provided by Huggingface’s transfomers library:

python

python -m transformers.onnx --model=distilbert-base-uncased-finetuned-sst-2-english --feature=sequence-classification onnx/ --opset 17

This command will output onnx/model.onnx file. Let’s run it locally first, to check if it’s working:

python

import onnxruntime as rt    
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')
sess = rt.InferenceSession("./onnx/model.onnx")
text = ['This is a positive tweet!', "Let's put some hate into this text", "Hello world!"]
inputs = tokenizer(text, return_tensors='np', padding=True)
output_name = sess.get_outputs()[0].name
output = sess.run([output_name], dict(inputs))
output[0].argmax(1)
# Output:
# array([1, 0, 1])

It works fine, but it will not work on BigQuery though…

⚠️ Limitation #2
BigQuery ML inference does not support multiple dynamic axes in ONNX models, the only dynamic axis can be the one responsible for batch size (which means - the first one).
State for 2023-05-14

Dynamic axes in ONNX allow to pass tensors of variable-lenght into the models - this is especially useful when you want to pass batches of various sizes into the model, instead of having them fixed to 1, 2,… etc. Models converted to ONNX with Huggingface’s transformers.onnx module have 2 dynamic axes - for NLP models it means: batch size and input sequence length. This is visible if you inspect the model:

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0]
# Output:
name: "input_ids"
type {
  tensor_type {
    elem_type: 7
    shape {
      dim {
        dim_param: "batch" # <--- first dynamic axis
      }
      dim {
        dim_param: "sequence" # <--- second dynamic axis
      }
    }
  }
}

Trying to deploy such model in BigQuery will result in the following error:

ONNX Model input ‘input_ids’ has unknown tensor dimension at index: 1, which is not supported.

How to fix it? - take 1

The model I’m using - DistilBERT - was trained with max_position_embeddings config set to 512, so let’s first try to set this value as a fixed size of the second axis in the ONNX’s models input (spoiler alert - it will not work).

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0].type.tensor_type.shape.dim[1].dim_value = 512 # <--- for input_ids
model.graph.input[1].type.tensor_type.shape.dim[1].dim_value = 512 # <--- for attention_mask
onnx.save(model, "./onnx/model_fixed.onnx")

Now, when I import the model into BigQuery it will still complain, this time - about something else:

ONNX model cannot be parsed within the memory limit; try reducing the model size

This happens, because ONNX needs to pre-allocate some memory for the input tensors, which also affect internal layer sizes, resulting in exhausted memory limit on the BigQuery side.

How to fix it? - take 2

I had to shrink the model inputs even further, down to 256 (which is still fine for short tweets)

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0].type.tensor_type.shape.dim[1].dim_value = 256 # <--- for input_ids
model.graph.input[1].type.tensor_type.shape.dim[1].dim_value = 256 # <--- for attention_mask
onnx.save(model, "./onnx/model_fixed_take_2.onnx")

Now it passes, BigQuery is able to create my model.

Creating BigQuery ONNX model

In order to import ONNX model into BigQuery inference engine, first upload the model (the fixed one :) ) to GCS:

bash

gsutil cp ./onnx/model_fixed_take_2.onnx gs://<your bucket>/models/distilbert-base-uncased-finetuned-sst-2-english.onnx

Then, open BigQuery UI and execute the following query:

sql

CREATE OR REPLACE MODEL
onnx_demo.sentiment_classifier
OPTIONS(MODEL_TYPE = 'ONNX', MODEL_PATH = "gs://<your bucket>/models/distilbert-base-uncased-finetuned-sst-2-english.onnx");

Once executed, you should see the model in the UI:

👀 Spoiler alert - you will have to fix one more thing in the ONNX model, follow the guide below.

Huggingface tokenizers in BigQuery

Wait… The model accepts input_ids and attention_mask, but our BigQuery table has text 😱 We need to tokenize the text first, before we pass it to the model, but tokenizers are Python code that cannot be easily converted to ONNX format. How to solve this problem? Use BigQuery Remote Functions!

Huggingface tokenizer as BigQuery Remote Function

BigQuery allows to create a UDF that can actually call either Cloud Functions or Cloud Run. Let’s leverage that. For more details also see my previous blog post here.

Full code of the Cloud Function:

python

import json
from transformers import AutoTokenizer
import functions_framework
import os

tokenizer_name = os.environ.get("TOKENIZER_NAME", "distilbert-base-uncased-finetuned-sst-2-english")
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)

def expand(input_dict):
    input_ids = input_dict["input_ids"]
    attention_mask = input_dict["attention_mask"]
    result = []
    for i in range(len(input_ids)):
        obj = {"input_ids": input_ids[i], "attention_mask": attention_mask[i]}
        result.append(obj)
    return result

@functions_framework.http
def tokenize_text(request):
    request_json = request.get_json()
    calls = request_json["calls"]
    texts = [row[0] for row in calls]
    tokenized = tokenizer(texts, truncation=True, max_length=256, padding="max_length")
    result = expand(tokenized)
    return json.dumps({"replies": [json.dumps(r) for r in result]}) # double serialization, because we need to return STR in Remote Function

First, I’m using functions_framework to be able to test my code locally. Secondly, I’m pre-loading distilbert-base-uncased-finetuned-sst-2-english on function start. Every call to the deployed function will be in BigQuery format, which looks like this:

python

{
  "request_id": "string",
  "caller": "string",
  "sessionUser": "string",
  "userDefinedContext": {},
  "calls": [
    ["values", "from", "row", 1],
    ["values", "from", "row", 2]
  ]
}

I’m invoking the tokenizer for the whole batch and then expand it from nested list into flat list of objects - each row will be in exact form that the ONNX model expects - it needs to have input_ids and attention_ids fields. Finally, the whole response is serialized, note the double serialization of JSON objects in each row. Right now BigQuery Remote Function does not support returning complex types, so I will just de-serialize this JSON afterwards in BigQuery SQL.

Make sure that you create requirements.txt file with all dependencies:

python

transformers~=4.29.1
functions-framework

Deploy the function using CLI:

bash

gcloud functions deploy tokenize_text --runtime python311 --trigger-http --min-instances 0 --no-allow-unauthenticated --region us-east1

Then, create connection between BigQuery and Cloud Functions and assing the Cloud Function Invoker role to the service account in this connection:

python

# 1. Create connection:
bq mk --connection --location=US --project_id=<project id> \
    --connection_type=CLOUD_RESOURCE huggingface-tokenizer
# 2. Copy the serviceAccountId from the results of:
bq show --format=prettyjson --connection <project id>.US.huggingface-tokenizer
# 3. Add Cloud Function Invoker role
gcloud functions add-iam-policy-binding tokenize_text --member="serviceAccount:bqcx-<redacted>@gcp-sa-bigquery-condel.iam.gserviceaccount.com" --role="roles/cloudfunctions.invoker" --region=us-east1

Once completed, add the remote function…

sql

CREATE OR REPLACE FUNCTION `onnx_demo`.tokenize(text STRING) RETURNS STRING
REMOTE WITH CONNECTION `<connection id>`
OPTIONS (
  endpoint = 'https://<cloud function url>.cloudfunctions.net/tokenize_text',
  max_batching_rows = 10 -- <-- adjust this depending on the memory limits of your Cloud Function
)

…and invoke it:

sql

select `onnx_demo`.tokenize(text) as t, text, label
from `onnx_demo.twitter_sentiment_validation` 
LIMIT 100

Result:

Great, we have deployed model and invoked the tokenizer, let’s put everything together!

DistilBERT inference on BigQuery

For clarity, I’ve split the inference code into 4 parts (CTEs):

Tokenizing the text
Transforming the tokenized JSONs into BigQuery arrays
Running the inference using the ONNX model
Parsing results to determine the text sentiment

Let’s start with #1, this query is simple - it’s just calling the tokenizer. Note that I’m explicitly limiting to 10 rows, see below why 👀

sql

with tokenized as (
  select `onnx_demo`.tokenize(text) as t, text, label
  from `onnx_demo.twitter_sentiment_validation` 
  LIMIT 10
)

Next part, #2 is parsing the JSONs from the tokenizer into BigQuery ARRAY<INT64> format:

sql

expanded as (
  select 
  (SELECT ARRAY_AGG(CAST(i AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.input_ids')) AS i) as input_ids, 
  (SELECT ARRAY_AGG(CAST(a AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.attention_mask')) AS a) as attention_mask, 
  text, label
  from tokenized
)

Calling the model (#3), while also passing the text and labels from the original table:

sql

predictions as (
  SELECT
    * EXCEPT(input_ids, attention_mask)
  FROM
    ML.PREDICT(MODEL `onnx_demo.sentiment_classifier`, (
      SELECT
        input_ids,
        attention_mask,
        text,
        label
      FROM
        expanded
      )
    )
)

Finally #4, parse the results (logits) from the model (if you have more than 2 classes in the model, you will effectively have to implement something similar to argmax to get the class predicted by the model).

sql

select text, (CASE WHEN logits[OFFSET(0)] > logits[OFFSET(1)] THEN 'negative' ELSE 'positive' END) as predicted, label, logits
from predictions

See the full query ⬇️

sql

with tokenized as (
  select `onnx_demo`.tokenize(text) as t, text, label
  from `onnx_demo.twitter_sentiment_validation` 
  LIMIT 10
), expanded as (
  select 
  (SELECT ARRAY_AGG(CAST(i AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.input_ids')) AS i) as input_ids, 
  (SELECT ARRAY_AGG(CAST(a AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.attention_mask')) AS a) as attention_mask, 
  text, label
  from tokenized
), predictions as (
  SELECT
    * EXCEPT(input_ids, attention_mask)
  FROM
    ML.PREDICT(MODEL `onnx_demo.sentiment_classifier`, (
      SELECT
        input_ids,
        attention_mask,
        text,
        label
      FROM
        expanded
      )
    )
)
select text, (CASE WHEN logits[OFFSET(0)] > logits[OFFSET(1)] THEN 'negative' ELSE 'positive' END) as predicted, label, logits
from predictions

Result:

Scalling the inference

In the paragraph above, I’ve deliberately put a LIMIT 10 in the initial query, to limit the number of rows. Why? Try to run the query without the limit and see what happens:

Resources exceeded during query execution: UDF out of memory

⚠️ Limitation #3
BigQuery ML inference does not allow to set the batch size during model creation, which makes scalling up the inference challenging - you effectively don’t know how many rows will be send to the ONNX model at runtime, which might very often results in UDF out of memory exception.
State for 2023-05-14

Fortunately, there is a workaround for that. Remember, that original ONNX exported model had 2 dynamic axes - first one being batch_size and second one - sequence length. We’ve first reduced the second one to a fixed value of 256. Now it’s time to set the batch size to 1, which will force the BigQuery to only send one input example at a time to the model, making the inference possible for any number of rows. 💡 If you have any suggestion how it can be fixed in any other way, let me know in comments below.

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0].type.tensor_type.shape.dim[0].dim_value = 1 # <--- force batch size to 1
model.graph.input[0].type.tensor_type.shape.dim[1].dim_value = 256
model.graph.input[1].type.tensor_type.shape.dim[0].dim_value = 1 # <--- force batch size to 1
model.graph.input[1].type.tensor_type.shape.dim[1].dim_value = 256
onnx.save(model, "./onnx/model_fixed_final.onnx")

Once you upload the model to GCS and re-create it using CREATE OR REPLACE MODEL, the inference should start to work at larger scale! 🎉

ONNX models vs Remote Functions

As you’ve seen, the ONNX feature is still in preview stage (Pre-GA), with some important limitations that you should be aware of. The biggest of them is the model size limit, making it impossible to deploy even medium size models, like base versions of BERT/RoBERTa. At the same time, a lot of data that customers store in BigQuery is tabular and those limits might be enough to deploy smaller models, e.g. TabNet model family. The API for ONNX models is also very simple, which makes it easy to use.

For NLP models that also require tokenization before doing the inference, Remote Functions are still a crucial building block. They allow to virtually run any Python code on the BigQuery side, meaning that you can use any Python library. The downside of this approach is that you need to write some code and deploy it separately as either Cloud Functions or Cloud Run.

Using Remote Functions for everything right now seems to have less limitations than ONNX models - you can deploy models on instances with up to 8 CPU and 32GB RAM (preview in Cloud Run), which will fit much larger models, with possibility of using quantized versions too.

If this post will get enough interest, I’ll be happy to prepare a side-by-side performance comparison of both approaches 📊.

Summary

I hope that this post helped you to deploy your own custom model in BigQuery ML. You also know the current limitations of ONNX models in BigQuery ML inference engine. Let me know what you think in the comments!

Additional links & resources

BigQuery Remote Functions documentation: https://cloud.google.com/bigquery/docs/remote-functions
BigQuery ML ONNX limitations: https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-onnx#limitations
BigQuery JSON functions: https://cloud.google.com/bigquery/docs/reference/standard-sql/json_functions

Deploy MLflow models on BigQuery

Marcin Zabłocki — Sun, 02 Apr 2023 01:00:00 +0000

Deploying machine learning models and making them easily accessible by other teams (especially data/business analysts) can be challenging. The landscape of MLOps tools in the recent months has exploded and opened a lot of opportunities for engineers (ML Engieeers / MLOps and you) to play with various systems and stich them together in creative and reliable way.

In this blogpost I will show you how you can easily deploy MLflow models to GCP Cloud Run service in a way that they could be consumed from BigQuery using SQL. I will achieve it by leveraging BigQuery Remote Functions feature.

💡 Recently (March 29, 2023) Google has announced a way to deploy models directly in BigQuery, I’ve also made a tutorial for this option here ➡️.
This blogpost shows an alternative option, which comes in handy, especially when you have custom logic / custom pre/post processing code in the models, that is not easily converted to ONNX format.

TL;DR

You will learn how to deploy MLflow models on Cloud Run & use them on BigQuery data using SQL. The whole project is available on GitHub (links below).

⚠️ What is not in the scope of this blogpost

I will not focus on the deployment of the MLflow instances here. Also, the model I will be deploying will be trained using TPOT’s AutoML for simplicity, as the Data Science part of the project is not important here.

What you will find below

Prerequisites

You will need:

Python (I’m using 3.9)
Docker
access to Google Cloud Platform (BigQuery & Cloud Run)
Artifact Registry for Docker / Google Container Registry available

Training a simple model with TPOT’s AutoML

As I’ve mentioned above, I will be not focusing on trainign the best-in-class model for the dataset, so for the simplicity, I’m using AutoML capabilities provided by the TPOT library.

The library has really simple API and the general workflow with it looks like this:

Load the data.

Create a classification or regression task

Set AutoML parameters, especially: number of generations, population size and paralleization level (they affect training time, usually - the longer = the better model)

Export the training pipeline - TPOT exports the scratch version of the training pipeline, that you can modify or copy to your existing project.

Modify the scratch version of the pipeline to load the data, train the model and save it somewhere.

For the save it somewhere part I will be using MLflow as it handles model serialization well: it not only saves (pickles) the trained model but also captures input data schema and input examples - this is really important from both usability and MLOps perspective, as you always know what to “send” to the model during inference.

In this blogpost, I will be using California Housing dataset (available directly from Scikit Learn).

Open requirements.txt

python

cloudpickle==2.2.1
fastapi==0.92.0
uvicorn==0.21.0
mlflow
scikit-learn==1.2.1
pandas==1.5.3
tpot==0.11.7
numpy==1.23.5
scipy==1.10.1

python

from functools import partial, update_wrapper
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from tpot import TPOTRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_california_housing


def main():
    # split into train and test
    dataset = fetch_california_housing(as_frame=True)
    dataset.target = dataset.target * 100000
    X_train, X_test, y_train, y_test = train_test_split(
        dataset.data.astype(float),
        dataset.target,
        train_size=0.7,
        test_size=0.3,
    )

    for ds, name in zip(
        (X_train, X_test, y_train, y_test), ("X_train", "X_test", "y_train", "y_test")
    ):
        ds.to_csv(f"{name}.csv", index=False)

    # create and fit TPOT
    tpot = TPOTRegressor(
        generations=10,
        population_size=50,
        verbosity=2,
        random_state=666,
        n_jobs=-1,
        scoring="neg_mean_absolute_error",
        config_dict="TPOT light",
    )
    tpot.fit(X_train, y_train)
    # calculate mean_absolute_error, means_squared_error, r2_score in a loop
    predictions = tpot.predict(X_test)
    for metric in [
        mean_absolute_error,
        update_wrapper(partial(mean_squared_error, squared=False), mean_squared_error),
        r2_score,
    ]:
        print(f"{metric.__name__}: {metric(y_test, predictions)}")
    tpot.export("tpot_pipeline_TO_EDIT.py")


if __name__ == "__main__":
    main()

After running the code above, you will see tpot_pipeline_TO_EDIT.py file in your working directory. It contains the best model that TPOT’s AutoML has found.

I’ve copied this file as tpot_pipeline.py and edited it by adding:

loading of my data
saving the model in the MLflow format.

python

import mlflow
import pandas as pd
from mlflow.models import infer_signature
from sklearn.feature_selection import VarianceThreshold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.svm import LinearSVR
from sklearn.tree import DecisionTreeRegressor
from tpot.builtins import StackingEstimator
from tpot.export_utils import set_param_recursive

X_train = pd.read_csv("X_train.csv")
y_train = pd.read_csv("y_train.csv")


with mlflow.start_run() as run:
    # Average CV score on the training set was: -39252.13134203072
    exported_pipeline = make_pipeline(
        RobustScaler(),
        VarianceThreshold(threshold=0.0001),
        StackingEstimator(
            estimator=LinearSVR(
                C=10.0,
                dual=True,
                epsilon=0.1,
                loss="squared_epsilon_insensitive",
                tol=0.001,
            )
        ),
        DecisionTreeRegressor(max_depth=10, min_samples_leaf=14, min_samples_split=4),
    )
    # Fix random state for all the steps in exported pipeline
    set_param_recursive(exported_pipeline.steps, "random_state", 666)

    exported_pipeline.fit(X_train, y_train)

    test_df = pd.read_csv("X_test.csv")
    mlflow.sklearn.save_model(
        exported_pipeline,
        "model",
        signature=infer_signature(X_train),
        input_example=test_df.sample(n=7, random_state=666),
    )
    print("Done :)")

Once you run this script, the “best” model will be trained and saved on disk in the model folder. MLflow’s model format is described here.

Creating BigQuery Remote Function with Fast API

First, let me explain how the BigQuery Remote Functions work: you create either a Cloud Function or Cloud Run endpoint, which needs to implement a contract defined in the documentation. Then, you create a Cloud resource connection in BigQuery - this will create a Service Account, that BQ will use to access the Cloud Function / Cloud Run service. After setting appropriate IAM to this service account, you create a definition for the remote function within BQ, specifying its signature, endpoint URL and options. Once everything is set, you will be able to run the Remote Function just like any other function in BigQuery - directly from SQL!

BigQuery Remote Function contract

BigQuery has a well defined format of the requests its sending to any Remote Function. An example request looks like this:

javascript

{
 "requestId": "124ab1c",
 "caller": "//bigquery.googleapis.com/projects/myproject/jobs/myproject:US.bquxjob_5b4c112c_17961fafeaf",
 "sessionUser": "test-user@test-company.com",
 "userDefinedContext": {
  "key1": "value1",
  "key2": "v2" // <--- any metadata you like to attach (it's defined when you register the function in BQ)
 },
 "calls": [
  [null, 1, "", "abc"], // <--- first row from the request
  ["abc", "9007199254740993", null, null] // <--- second row from the request
 ]
}

In order to implement a FastAPI endpoint accepting requests in the format shown above, let’s create a pydantic dataclass for it:

python

from typing import Dict, List, Any, Optional
from pydantic import BaseModel

class BigQueryUDFRequest(BaseModel):
    request_id: Optional[str]
    caller: str
    sessionUser: str
    userDefinedContext: Optional[Dict]
    calls: List[List[Any]] = []

The response contract is simpler, it only needs a single replies field, as shown below.

python

class BigQueryUDFResponse(BaseModel):
    replies: List[Any]

Last thing - BigQuery Remote Functions only support root level endpoints - that means, that the endpoint in the FastAPI must accept POST requests on the / (root) path. No nested paths are allowed at the moment of writing this blogpost.

Remote Function endpoint

Definition of an endpoint and the FastAPI appplication looks like this:

python

import mlflow.sklearn
import pandas as pd
from fastapi import FastAPI
from api.models import BigQueryUDFRequest, BigQueryUDFResponse, AppContext

app = FastAPI()
ctx = AppContext()

@app.on_event("startup")
def load_model():
    # Load your MLflow model here
    ctx.model = mlflow.sklearn.load_model("model")

@app.post("/", response_model=BigQueryUDFResponse)
def udf(request: BigQueryUDFRequest):
    df = pd.DataFrame(request.calls)
    # Optionally - parse the request, do some logging etc
    predictions = ctx.model.predict(df)
    return BigQueryUDFResponse(replies=predictions.tolist())

The AppContext is defined like this:

python

@dataclass
class AppContext:
    model: Any = None

Testing out the endpoint locally

Once you run the app locally, you can either open http://localhost:8000/docs and send requests from Swagger UI, or use curl. For my AutoML model, the request looks like this:

bash

curl -X 'POST' \
  'http://localhost:8000/' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "request_id": "string",
  "caller": "string",
  "sessionUser": "string",
  "userDefinedContext": {},
  "calls": [[3.1779, 16.0, 4.636165577342048, 0.9607843137254902, 1860.0, 4.052287581699346, 38.04, -121.29], 
            [4.1364, 24.0, 23.54736842105263, 5.168421052631579, 264.0, 2.778947368421053, 39.27, -120.04]]
}'

As you can see, batch requests are possible out-of-the-box.

Deploying Remote Function to Cloud Run

Building a Docker container

First, a docker container with our custom function needs to be build.

For tutorial purposes, the MLflow model will be copied into the container itself. In production scenarios, you will most likely load the model on the fly from GCS during the container startup process.

An ENTRYPOINT for Cloud Run and Fast API can look like this (save this as run.sh). It’s important to dyunamically use the $PORT environment variable, as it will be set by the Cloud Run’s runtime.

python

#!/bin/bash
set -e
exec uvicorn bq_api:app --proxy-headers --host 0.0.0.0 --port $PORT

… and the Dockerfile itself:

python

FROM python:3.9.16-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt --no-cache-dir
COPY . .
RUN chmod +x ./run.sh
ENV PORT 8000
ENTRYPOINT ["./run.sh"]

Build the docker image.

python

export IMAGE="us-east1-docker.pkg.dev/<full path to artifact registry>/mlflow-bq-example:20220401-1942"

# On Macs with M1/M2 processor:
docker buildx build --platform=linux/amd64 -t $IMAGE .

# On Linux/other:
docker build -t $IMAGE .

Now, push the image to the Artifact Registry / Container Registry (docker push $IMAGE).

Create Cloud Run service

With GCLOUD CLI, deployment to Cloud Run is a one-liner. The command provides sensible defaults for the tutorial purposes.

bash

gcloud run deploy mlflow-bigquery \
    --cpu 1 --memory 1Gi --image=$IMAGE \
    --region=us-east1 --no-allow-unauthenticated

after a few seconds, the deployment should finish:

bash

Deploying container to Cloud Run service [mlflow-bigquery] in project [<project>] region [us-east1]
✓ Deploying new service... Done.                                                                                                                                                                        
  ✓ Creating Revision...                                                                                                                                                                                
  ✓ Routing traffic...                                                                                                                                                                                  
Done.                                                                                                                                                                                                   
Service [mlflow-bigquery] revision [mlflow-bigquery-00001-yav] has been deployed and is serving 100 percent of traffic.
Service URL: https://mlflow-bigquery-xyz123-ue.a.run.app

Thanks to the --no-allow-unauthenticated, the Cloud Run service will be protected from unauthorized access by default. In order to send an authorized requests to the Cloud Run service, use the following command:

bash

curl -X 'POST' \
  'https://mlflow-bigquery-xyz123-ue.a.run.app' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -d '{
  "request_id": "string",
  "caller": "string",
  "sessionUser": "string",
  "userDefinedContext": {},
  "calls": [[3.1779, 16.0, 4.636165577342048, 0.9607843137254902, 1860.0, 4.052287581699346, 38.04, -121.29], 
            [4.1364, 24.0, 23.54736842105263, 5.168421052631579, 264.0, 2.778947368421053, 39.27, -120.04]]
}'

result should be exactly the same as before.

Connecting from BigQuery to Cloud Run

⚠️ Before proceeding, make sure that you have BigQuery Connection API enabled in your GCP project.

You can either configure the connection from UI or use command line.

(option 1) From UI

Click Add

Click Connections to external data sources
Fill in the form, in the Connection type list, use BigLake and remote functions (Cloud Resource).

Click Create connection button.
In the left pane, open External connections and view the created connection details. Copy Service account id for later use.

(option 2) From CLI

bash

bq mk --connection --location=US --project_id=<project id> \
    --connection_type=CLOUD_RESOURCE mlflow-cloudrun-cli

javascript

bq show --format=prettyjson --connection <project id>.US.mlflow-cloudrun-cli
{
  "cloudResource": {
    "serviceAccountId": "bqcx-<redacted>@gcp-sa-bigquery-condel.iam.gserviceaccount.com"
  },
  "creationTime": "1680373492742",
  "lastModifiedTime": "1680373492742",
  "name": "projects/<redacted>/locations/us/connections/mlflow-cloudrun-cli"
}

Once the External connection and service account id is created, we need to allow this service to access Cloud Run.

bash

gcloud run services add-iam-policy-binding mlflow-bigquery \
  --member='serviceAccount:bqcx-<redacted>@gcp-sa-bigquery-condel.iam.gserviceaccount.com' \
  --role='roles/run.invoker' --region=us-east1

Running the inference from BigQuery

Now it’s time to create the actual function to invoke from BigQuery. Open query editor and create function definition similar to the one below. Make sure that you set the number of parameters right.

💡 The max_batching_rows = 128 parameter is really important as it affects performance of the inference - it effectively tells BigQuery, how many rows of the input table to send to Cloud Run in a single request. Setting the max_batching_rows as well as the concurrency in the Cloud Run (max requests per container) allows to fine-tune the inference performance.

sql

CREATE OR REPLACE FUNCTION `<project id>.mlflow_model_demo`.predict(
  MedInc FLOAT64,
  HouseAge FLOAT64,
  AveRooms FLOAT64,
  AveBedrms FLOAT64,
  Population FLOAT64,
  AveOccup FLOAT64,
  Latitude FLOAT64,
  Longitude FLOAT64
) RETURNS FLOAT64
REMOTE WITH CONNECTION `<project id>.US.mlflow-bigquery-cloudrun`
OPTIONS (
  endpoint = 'https://mlflow-bigquery-xyz123-ue.a.run.app'
  max_batching_rows = 128
)

The above statement will create a new custom function named predict in the mlflow_model_demo dataset. After successful run, the following message should display in the query results panel:

bash

This statement created a new function named <project id>.mlflow_model_demo.predict.

Once the function is created, let’s use it!

sql

SELECT *,
`<project id>.mlflow_model_demo`.predict(
    MedInc,
    HouseAge,
    AveRooms,
    AveBedrms,
    Population,
    AveOccup,
    Latitude,
    Longitude
) as price,
 FROM `<project id>.mlflow_model_demo.california_housing_test` LIMIT 1000

after a few seconds, results will display:

In the Cloud Run dashboard, you will be able to observe the performance metrics - they will be an useful input for setting the max_batching_rows in BQ and concurrency in Cloud Run to achieve optimal ML model inference performance in this setup.

💡Pro-tips

Here’s a bunch of pr0-tips useful while working with this setup:

MLflow model saving

When you save the model in MLflow, use input_example parameter of save_model- it will give you a JSON with exact data shape and types that your model accepts.
Also in MLflow, use mlflow.models.signature.infer_signature and signature param of save_model to capture metadata about data types - also useful for future debugging.

Multiple Remote Functions in a single endpoint

You actually CAN have multiple BigQuery Remote Functions deployed in a single Cloud Run Service. In order to do this, you need to specify user_defined_context during function create and handle routing in the root endpoint on your own.

See the code

sql

CREATE OR REPLACE FUNCTION `<project id>.mlflow_model_demo`.other_function(
  col1 FLOAT64,
  col1 FLOAT64,
) RETURNS FLOAT64
REMOTE WITH CONNECTION `<project id>.US.mlflow-bigquery-cloudrun`
OPTIONS (
  user_defined_context = [("function", "other-function")]
)

Then, in the function handler in FastAPI do the routing on the contents of request.userDefinedContext dictionary

python

@app.post("/", response_model=BigQueryUDFResponse)
def udf(request: BigQueryUDFRequest):
    df = pd.DataFrame(request.calls)
    if request.userDefinedContext["function"] == "predict":
        results = ctx.model.predict(df)
    elif request.userDefinedContext["function"] == "other-function":
        results = call.other.function()
    return BigQueryUDFResponse(replies=results.tolist())

Summary

I hope that this post helped you to deploy any custom MLflow model to Cloud Run and use it directly from BigQuery just like any other SQL function.

Additional links & resources

GitHub repo with full project - https://github.com/marrrcin/mlflow-bigquery-remote-function
Deploying secure MLflow on App Engine: https://getindata.com/blog/deploying-mlflow-google-cloud-platform-using-app-engine/
Deploying secure MLflow on Cloud Run: https://getindata.com/blog/deploying-serverless-mlflow-google-cloud-platform-using-cloud-run/

Robust Subspace Recovery Layer implementation in PyTorch

Marcin Zabłocki — Mon, 19 Apr 2021 00:00:00 +0200

Anomaly detection (or outlier detection) is one of the many tasks that keep Data Scientists awake at night when then need to satisfy the business needs. No matter whether the data in well structured (like user events) or unstructured (like text or images), there are plenty of methods that allow to tackle this problem and it’s diffictult to choose one. Sometimes, it’s even difficult to properly define, what an anomaly is - but that’s not the part of this post’s main focus. Some of the most popular methods for anomaly detection are:

statistical methods - like three-sigma rule and regression models
classical machine learning methods - like one-class SVM, Isolation Forests or clustering-based methods (e.g. kNN)
deep learning methods - like autoencoders and deep forecasting models (e.g. LSTMs for time series)

In this post I will focus on a single method called Robust Subspace Recovery Layer for Unsupervised Anomaly Detection (https://arxiv.org/abs/1904.00152) which is an extension of the autoencoder-based unsupervised anomnaly detection method for data of any domain.

TL;DR

Implementation of RSR Layer and RSR Autoencoder (RSRAE) in PyTorch. Link to Jupyter Notebook / Google Colab at the bottom.

Note: This post and my implementation does not reproduce RSR Layer paper. I only explain the general idea and provide minimal working code as a form of proof-of-concept.

Idea behind Robust Space Recovery Layer

Autoencoders were used for the anomaly detection much earlier than this (RSR) paper appeared. To use autoencoder in a setting of anomaly detection, one treats “normal” examples (inliers) as the ones that have low reconstruction error and other which have high reconstruction error are treated as outliers. Although the idea is fairly easy to understand and implement, the problem lies in the fact that in order for autoencoder to work properly in this setting (i.e. to detect the outliers) the training data should be clean (or have really small fraction of the outliers). In the real world it’s highly unlikely for the input data to be free of any anomalies and even if Data Scientists could label and filter the data (given they know what an anomaly in the given dataset is) it simply won’t scale.

The Robust Subspace Recovery Layer paper tries to adress this issue by incorporating the RSR Layer into autoencoders. It’s clearly stated that this method does not compete with semi-supervised or supervised anomaly detection setting, but shows its strenght in the unsupervised setting.

What is the RSR Layer?

The RSR Layer itself is just an additional layer in the nerual network with specific properties. It maps one representation (embedding obtained from the encoder) into another low-dimensional representation that is outlier-robust.

Let’s say that D (upper case D) is the dimension of the embedding from the encoder. The assumption in the paper is that the “normal” data lies within d-dimensional (lower case d) manifold (“subspace”) of the original embedding, which means that d < D.

Please refer to the paper to obtain more details and the full explanation and analysis. At this point you should know enough to proceed to the implementation section.

How to implement the RSR Layer in PyTorch?

As RSR Layer is just a linear mapping, there are only a few things needed:

The dimension D from the encoder’s output.
The dimension d which is a direct property of the RSR Layer (and probably requires a hyperparamer search to set it properly).
Proper initialization of the linear mapping weights (the optimization goal of the RSR Autoencoder enforces that, so it’s better to start with a set of weights that already fulfill this constraint).

The implementation of the layer itself is straightforward.

python

import torch
from torch import nn
import torch.nn.functional as F

python

class RSRLayer(nn.Module):
    def __init__(self, d:int, D: int):
        super().__init__()
        self.d = d
        self.D = D
        self.A = nn.Parameter(torch.nn.init.orthogonal_(torch.empty(d, D)))

    def forward(self, z):
        # z is the output from the encoder
        z_hat = self.A @ z.view(z.size(0), self.D, 1)
        return z_hat.squeeze(2)

The next piece to obtain RSR Autoencoder in PyTorch is to implement RSR Loss as per paper’s equation (4): The first term enforces the RSR Layer projection to be robust and the second term enforces the projection to be orthogonal. Refer to the paper for the motivation behind this loss as it it out of scope for this blog post.

Nearly op-for-op (math-to-PyTorch) implementation of this loss looks like this:

python

class RSRLoss(nn.Module):
    def __init__(self, lambda1, lambda2, d, D):
        super().__init__()
        self.lambda1 = lambda1
        self.lambda2 = lambda2
        self.d = d
        self.D = D
        self.register_buffer(
            "Id", torch.eye(d)
        )

    def forward(self, z, A):
        z_hat = A @ z.view(z.size(0), self.D, 1)
        AtAz = (A.T @ z_hat).squeeze(2)
        term1 = torch.sum(
            torch.norm(z - AtAz, p=2)
        )

        term2 = torch.norm(
            A @ A.T - self.Id, p=2
        ) ** 2

        return self.lambda1 * term1 + self.lambda2 * term2

Lastly, the standard L1/L2 reconstruction loss is replaced by the L_2,p-norm based loss (again, the paper relies on “more robustness”, wiki: https://en.wikipedia.org/wiki/Matrix_norm#L2,1_and_Lp,q_norms). In practice, the paper uses L_2,1 version.

python

class L2p_Loss(nn.Module):
    def __init__(self, p=1.0):
        super().__init__()
        self.p = p
  
    def forward(self, y_hat, y):
        return torch.sum(
            torch.pow(
                torch.norm(y - y_hat, p=2), self.p
            )
        )

RSR Auto Encoder in PyTorch

In order to see the RSR Layer in practice, the RSR Autoencoder needs to be implemented.

Important
The following RSR Autoencoder implementation is only a minimal, working proof of concept. As I’ve stated earlier, the intent of this blog post is to implement the RSR Layers and play around with them, not reach the SOTA or reproduce the paper’s results. Feel free to use my code and plug-in your encoder and decoder networks.

python

class RSRAutoEncoder(nn.Module):
    def __init__(self, input_dim, d, D):
        super().__init__()
        # Put your encoder network here, remember about the output D-dimension
        self.encoder = nn.Sequential(
          nn.Linear(input_dim, input_dim // 2),
          nn.LeakyReLU(),
          nn.Linear(input_dim // 2, input_dim // 4),
          nn.LeakyReLU(),
          nn.Linear(input_dim // 4, D)
        )

        self.rsr = RSRLayer(d, D)

        # Put your decoder network here, rembember about the input d-dimension
        self.decoder = nn.Sequential(
          nn.Linear(d, D),
          nn.LeakyReLU(),
          nn.Linear(D, input_dim // 2),
          nn.LeakyReLU(),
          nn.Linear(input_dim // 2, input_dim)
        )
    
    def forward(self, x):
        enc = self.encoder(x) # obtain the embedding from the encoder
        latent = self.rsr(enc) # RSR manifold
        dec = self.decoder(latent) # obtain the representation in the input space
        return enc, dec, latent, self.rsr.A

Training the autoencoder with RSR Layer in PyTorch (& PyTorch Lightning)

As usual, I use PyTorch Lightning to skip the training boilerplate.

python

import pytorch_lightning as pl
pl.seed_everything(666)

Corrupted MNIST dataset (with outliers)

For this proof-of-concept I will be using a corrupted version of MNIST dataset, that contains single class as “normal” data and a few other, that will be treated as outliers.

python

from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor
# https://github.com/pytorch/vision/issues/1938#issuecomment-594623431
from six.moves import urllib
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)

mnist = MNIST(".", download=True, transform=ToTensor())

python

class RSRDs(torch.utils.data.Dataset):
    def __init__(self, target_class, other_classes, n_examples_per_other):
        super().__init__()
        self.mnist = MNIST(".", download=True, transform=ToTensor())
        self.target_indices = (self.mnist.targets == target_class).nonzero().flatten()
        
        other = []
        for other_class in other_classes:
            other.extend(
                (self.mnist.targets == other_class).nonzero().flatten()[:n_examples_per_other]
            )
        self.other_indices = torch.tensor(other)
        self.all_indices = torch.cat([self.other_indices, self.target_indices])
        print(f"Targets: {self.target_indices.size(0)}")
        print(f"Others : {self.other_indices.size(0)}")
    
    def __getitem__(self, idx):
        actual_idx = self.all_indices[idx].item()
        return self.mnist[actual_idx]
    
    def __len__(self):
        return self.all_indices.size(0)

python

ds = RSRDs(target_class=4, other_classes=(0, 1, 2, 8), n_examples_per_other=100)

Targets: 5842
Others : 400

python

ds[0][0].shape, ds[0][1]

(torch.Size([1, 28, 28]), 0)

RSRAutoencoder module in PyTorch Lightning

The following module combines all of the pieces together:

RSR Autoencoder with RSR Layer inside,
L2p reconstruction loss,
RSR Loss.

python

class RSRAE(pl.LightningModule):
    def __init__(self, hparams):
        super().__init__()
        self.hparams = hparams
        self.ae = RSRAutoEncoder(
            self.hparams.input_dim, 
            self.hparams.d, 
            self.hparams.D)
        self.reconstruction_loss = L2p_Loss(p=1.0)
        self.rsr_loss = RSRLoss(self.hparams.lambda1, self.hparams.lambda2, self.hparams.d, self.hparams.D)
  
    def forward(self, x):
        return self.ae(x)
  
    def training_step(self, batch, batch_idx):
        X, _ = batch
        x = X.view(X.size(0), -1)
        enc, dec, latent, A = self.ae(x)

        rec_loss = self.reconstruction_loss(torch.sigmoid(dec), x)
        rsr_loss = self.rsr_loss(enc, A)
        loss = rec_loss + rsr_loss
        
        # log some usefull stuff
        self.log("reconstruction_loss", rec_loss.item(), on_step=True, on_epoch=False, prog_bar=True)
        self.log("rsr_loss", rsr_loss.item(), on_step=True, on_epoch=False, prog_bar=True)
        return {"loss": loss}

    def configure_optimizers(self):
        opt = torch.optim.AdamW(self.parameters(), lr=self.hparams.lr)
        # Fast.AI's best practices :)
        scheduler = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=self.hparams.lr, 
                                                        epochs=self.hparams.epochs, 
                                                        steps_per_epoch=self.hparams.steps_per_epoch)
        return [opt], [{
            "scheduler": scheduler,
            "interval": "step"
        }]

python

dl = DataLoader(ds, batch_size=64, shuffle=True, drop_last=True)

python

hparams = dict(
    d=16,
    D=128,
    input_dim=28*28,
    # Peak learning rate
    lr=0.01,
    # Configuration for the OneCycleLR scheduler
    epochs=150,
    steps_per_epoch=len(dl),
    # lambda coefficients from RSR Loss
    lambda1=1.0,
    lambda2=1.0,
)
model = RSRAE(hparams)
model

python

RSRAE(
    (ae): RSRAutoEncoder(
    (encoder): Sequential(
        (0): Linear(in_features=784, out_features=392, bias=True)
        (1): LeakyReLU(negative_slope=0.01)
        (2): Linear(in_features=392, out_features=196, bias=True)
        (3): LeakyReLU(negative_slope=0.01)
        (4): Linear(in_features=196, out_features=128, bias=True)
    )
    (rsr): RSRLayer()
    (decoder): Sequential(
        (0): Linear(in_features=16, out_features=128, bias=True)
        (1): LeakyReLU(negative_slope=0.01)
        (2): Linear(in_features=128, out_features=392, bias=True)
        (3): LeakyReLU(negative_slope=0.01)
        (4): Linear(in_features=392, out_features=784, bias=True)
    )
    )
    (reconstruction_loss): L2p_Loss()
    (rsr_loss): RSRLoss()
)

python

trainer = pl.Trainer(max_epochs=model.hparams.epochs, gpus=1)
trainer.fit(model, dl)

Analysing RSR Autoencoder outputs

First, let’s check the general autoencoder reconstuction.

python

model.freeze()
from torchvision.transforms import functional as tvf
from torchvision.utils import make_grid
def reconstruct(x, model):
    enc, x_hat, latent, A = model(x.view(1, -1))
    x_hat = torch.sigmoid(x_hat)
    return tvf.to_pil_image(
      make_grid([x_hat.squeeze(0).view(1, 28, 28), x])
    )

python

tvf.to_pil_image(
    make_grid([tvf.to_tensor(reconstruct(ds[i][0], model)) for i in torch.randint(0, len(ds), (8, ))], nrow=1)
)

Most of the images seem OK to me, given how simple the model is.

Now let’s look at the RSR embedding space projected into 3D using PCA.

python

import pandas as pd
rsr_embeddings = []
classes = []
errors = []
for batch in iter(DataLoader(ds, batch_size=64, shuffle=False)):
    X, cl = batch
    x = X.view(X.size(0), -1)
    enc, x_hat, latent, A = model(x)
    rsr_embeddings.append(latent)
    classes.extend(cl.numpy())
    for i in range(X.size(0)):
        rec_error = L2p_Loss()(torch.sigmoid(x_hat[i]).unsqueeze(0), x[i].unsqueeze(0))
        errors.append(float(rec_error.numpy()))
        
all_embs = torch.vstack(rsr_embeddings)
df = pd.DataFrame(all_embs.numpy(), columns=["x", "y", "z"] + [f"dim_{i}" for i in range(hparams["d"] - 3)])
df.loc[:, "class"] = classes
df.loc[:, "errors"] = errors

python

from sklearn.decomposition import PCA
pca = PCA(n_components=3)
rsr_3d = pd.DataFrame(pca.fit_transform(all_embs), columns=["x", "y", "z"])
rsr_3d.loc[:, "class"] = classes

python

import plotly.express as px
import plotly
plotly.offline.init_notebook_mode()
df = df
fig = px.scatter_3d(df, x='x', y='y', z='z',
               symbol="class", color="class", opacity=0.95)
fig.update_layout(margin=dict(l=0, r=0, b=0, t=0))
fig.show()

Indeed, most of the 4 (the main class - inliers) reside in the center of the space, with other classes positioned further away from the center.

Unsupervised anomaly detection using RSR Autoencoder

Now let’s see if it works as the authors of the paper claim - whether the data points with high reconstruction error are really outliers.

python

df.errors.describe()

python

    count    6242.000000
    mean        1.551347
    std         0.293904
    min         0.615430
    25%         1.355003
    50%         1.512501
    75%         1.707990
    max         3.462465
    Name: errors, dtype: float64

python

df.groupby("class").errors.hist(legend=True, bins=60, figsize=(12, 12))

The reconstruction loss is centered around 1.5 with a tail spreading up to 3.46. What are the data points with the largest errors?

python

lowest_mistakes = df.sort_values(by="errors", ascending=True).head(60).loc[:, ["errors", "class"]]
highest_mistakes = df.sort_values(by="errors", ascending=False).head(60).loc[:, ["errors", "class"]]
highest_mistakes.head(10)

	errors	class
5917	3.462465	4
2201	3.120193	4
5834	3.116279	4
6017	3.109386	4
601	3.098328	4
983	3.097286	4
1312	3.091813	4
3304	3.030314	4
6015	2.998559	4
1619	2.974236	4

It seems really odd at first, as I've expected all the non-4 classes to have the largest errors. Let's carefully look at the actual results though:

python

print("Images with the highest reconsturction loss")
tvf.to_pil_image(
    make_grid([tvf.to_tensor(reconstruct(ds[i][0], model)) for i in highest_mistakes.index], nrow=6,
             pad_value=0.5)
)

Images with the highest reconsturction loss

python

print("Images with the lowest reconsturction loss")
tvf.to_pil_image(
    make_grid([tvf.to_tensor(reconstruct(ds[i][0], model)) for i in lowest_mistakes.index], nrow=6,
             pad_value=0.5)
)

Images with the lowest reconsturction loss

Now it’s much clearer - the model found the manifold that separates the images into the “normal” ones - that in this setting seem like more smooth ones with clean curves - and the outliers that are rather sharp, contain some random artifacts (which are anomalies IMHO) and usually seem slightly malformed. Given the model simplicity and the setup, it looks to me like a good starting point for the unsupervised anomaly detection.

Summary

I hope you find this post useful and it will help you to quickly get started with RSR Layer / RSRAE and unsupervised anomaly detection in PyTorch. Do not hesistate to comment and share!

Additional links & resources

Jupyter notebook for this post in Google Colab
GitHub repo with this notebook: https://github.com/marrrcin/rsrlayer-pytorch

13 features of PyTorch that you should know

Marcin Zabłocki — Sun, 21 Feb 2021 16:00:48 +0200

PyTorch has gained a lot of traction in both academia as well as in applied research in the industry. It’s a deep learning framework with great elasticity and huge number of utilities and functions to speed up the work. PyTorch’s learning curve is not that steep but implementing both efficient and clean code in it can be tricky. After using it for over 2 years, here are my top PyTorch features I wish I knew when I stared learning it.

TL;DR

13 features of PyTorch that you should know - a short list.

1. DatasetFolder

One of the first things people do when learning PyTorch is implementing their own Dataset of some kind. It’s a rookie mistake - there is no point of wasting time on writing such. Usually, datasets are either lists of data (or numpy arrays) or files on disk. It’s always better to have the data organized on a disk rather than writing custom Dataset to load the weird format that someone (or you) decided to store it with.

One of the most common data formats for classifiers is to have a directory with subfolders, representing classes, and files in those subfolders, representing examples, like the one below.

bash

folder/class_0/file1.txt
folder/class_0/file2.txt
folder/class_0/...

folder/class_1/file3.txt
folder/class_1/file4.txt

folder/class_2/file5.txt
folder/class_2/...

There is a built-in way to load this kind of datasets - does not matter if the data you have are images, text files or something else - just use DatasetFolder (https://pytorch.org/vision/0.8/datasets.html#datasetfolder). Surprisingly, this class is a part of torchvision package, not the core PyTorch. The class super comprehensive - you can filter files from the folder, load them using custom code and transform the raw files on the fly. Example:

python

from torchvision.datasets import DatasetFolder
from pathlib import Path
# I have text files in this folder
ds = DatasetFolder("/Users/marcin/Dev/tmp/my_text_dataset", 
    loader=lambda path: Path(path).read_text(),
    extensions=(".txt",), #only load .txt files
    transform=lambda text: text[:100], # only take first 100 characters
)

# Everything you need is already there
len(ds), ds.classes, ds.class_to_idx

python

(20, ['novels', 'thrillers'], {'novels': 0, 'thrillers': 1})

If you’re working with images, there is also a torchvision.datasets.ImageFolder class, which bases on the DatasetLoader and it’s preconfigured to load images.

2. Stop using `.to(device)` that much: `zeros_like` / `ones_like` etc.

I’ve read a lot fo PyTorch code from various GitHub repositories. What irritates me the most is that almost in every repo there is a lot of *.to(device) lines, which put the data from CPU to GPU or the other way around. Such lines usually occur in a lot of quick-and-dirty repos or beginner tutorials. I highly encourage of implementing such operations as little as possible and rely on the built-in PyTorch functionality that does that automatically. Putting those .to(device) lines here and there usually leads to either degraded performance or PyTorch-particioners’ favourite exception:

Expected object of device type cuda but got device type cpu

Obviously, there are some cases when you just can’t get around it, but most (if not all) of the trivial cases are covered. One of the examples is initialization of zeros/ones/random tensors that you might need in your custom loss - as the training of deep neural networks usually happens on GPU, model’s output is already on “cuda” device, but you now need to have another tensor of zeros/ones also be on “cuda” device to do operations on it. This is where *_like operations from PyTorch come in handy:

python

my_output # on any device, if it's cuda then my_zeros will also be on cuda
my_zeros = torch.zeros_like(my_output_from_model)

Under the hood, what PyTorch does is it invokes the following operation:

python

my_zeros = torch.zeros(my_output.size(), dtype=my_output.dtype, layout=my_output.layout, device=my_output.device)

so everything is set properly, and you decrease the probability of bugs in your code. Built-in operations include:

python

torch.zeros_like()
torch.ones_like()
torch.rand_like()
torch.randn_like()
torch.randint_like()
torch.empty_like()
torch.full_like()

3. Register Buffer (a.k.a `nn.Module.register_buffer`)

This is a next stop on my crusade to discourage people from using .to(device) everywhere. Sometimes your model or loss function needs to have parameters that are set upfront and are used when forward pass is invoked - for instance it can be a “weight” parameter, which scales the loss or some fixed tensor that is not changing but it’s used every time. For this kind of scenarios use nn.Module.register_buffer method, which tells PyTorch to store the values you pass to it within the module and to move those values with the module. If you initialize your module and then move it to GPU, those values will also be automatically moved. Moreover - if you save your module’s state - the buffers will also be saved!

Once registered, the values can be accessed in the forward function just like any other module’s property.

python

from torch import nn
import torch

class ModuleWithCustomValues(nn.Module):
    def __init__(self, weights, alpha):
        super().__init__()
        self.register_buffer("weights", torch.tensor(weights))
        self.register_buffer("alpha", torch.tensor(alpha))
    
    def forward(self, x):
        return x * self.weights + self.alpha

m = ModuleWithCustomValues(
    weights=[1.0, 2.0], alpha=1e-4
)
m(torch.tensor([1.23, 4.56]))

python

tensor([1.2301, 9.1201])

4. Built-in `Identity()`

Sometimes when you play with transfer learning you will need to replace some layers with 1:1 mapping, which boils down to implemeting a nn.Module with the only purpose or returning the input value. PyTorch has this class built-in: https://pytorch.org/docs/1.7.1/generated/torch.nn.Identity.html?highlight=identity#identity.

Example - you want to get image representation from a pre-trained ResNet50 just before the classification layer. Here is how to do this:

python

from torchvision.models import resnet50
model = resnet50(pretrained=True)
model.fc = nn.Identity()
last_layer_output = model(torch.rand((1, 3, 224, 224)))
last_layer_output.shape

python

torch.Size([1, 2048])

5. Pairwise distances: `torch.cdist`

The next time you will encounter a problem of calculating all-pairs euclidean (or in general: a p-norm) distance between two tensors, remember about torch.cdist. It does exactly that and also automatically uses matrix multiplication when euclidean distance is used, giving a performance boost.
Docs: https://pytorch.org/docs/1.7.1/generated/torch.cdist.html#torch-cdist

python

points1 = torch.tensor([[0.0, 0.0], [1.0, 1.0], [2.0, 2.0]])
points2 = torch.tensor([[0.0, 0.0], [-1.0, -1.0], [-2.0, -2.0], [-3.0, -3.0]]) # batches don't have to be equal
torch.cdist(points1, points2, p=2.0)

python

tensor([[0.0000, 1.4142, 2.8284, 4.2426],
        [1.4142, 2.8284, 4.2426, 5.6569],
        [2.8284, 4.2426, 5.6569, 7.0711]])

Performance of without and with matrix multiplications - it’s over 2x faster on my machine when using mm.

python

%%timeit
points1 = torch.rand((512, 2))
points2 = torch.rand((512, 2))
torch.cdist(points1, points2, p=2.0, compute_mode="donot_use_mm_for_euclid_dist")

867 µs ± 142 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

python

%%timeit
points1 = torch.rand((512, 2))
points2 = torch.rand((512, 2))
torch.cdist(points1, points2, p=2.0)

417 µs ± 52.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

6. Cosine similarity: `F.cosine_similarity`

Staying within the same topic as in the last point - calculating distances - euclidean distance is not always the thing you need. When working with vectors, usually the cosine similarity is the metric of choice. PyTorch has a built-in implementation of cosine similarity too.

python

import torch.nn.functional as F
vector1 = torch.tensor([0.0, 1.0])
vector2 = torch.tensor([0.05, 1.0])
print(F.cosine_similarity(vector1, vector2, dim=0))
vector3 = torch.tensor([0.0, -1.0])
print(F.cosine_similarity(vector1, vector3, dim=0))

python

tensor(0.9988)
tensor(-1.)

Batched cosine similarity in PyTorch

python

import torch.nn.functional as F
batch_of_vectors = torch.rand((4, 64))
similarity_matrix = F.cosine_similarity(batch_of_vectors.unsqueeze(1), batch_of_vectors.unsqueeze(0), dim=2)
similarity_matrix

python

tensor([[1.0000, 0.6922, 0.6480, 0.6789],
        [0.6922, 1.0000, 0.7143, 0.7172],
        [0.6480, 0.7143, 1.0000, 0.7312],
        [0.6789, 0.7172, 0.7312, 1.0000]])

7. Normalizing vectors: `F.normalize`

Last point still loosely connected to vectors and distances is normalization: usually it’s used to improve the stability of calculation, by changing the magnitude of the vector. The most commonly used normalization is L2 one and can be applied in PyTorch as follows:

python

vector = torch.tensor([99.0, -512.0, 123.0, 0.1, 6.66])
normalized_vector = F.normalize(vector, p=2.0, dim=0)
normalized_vector

python

tensor([ 1.8476e-01, -9.5552e-01,  2.2955e-01,  1.8662e-04,  1.2429e-02])

The old way of doing normalization in PyTorch was:

python

vector = torch.tensor([99.0, -512.0, 123.0, 0.1, 6.66])
normalized_vector = vector / torch.norm(vector, p=2.0)
normalized_vector

python

tensor([ 1.8476e-01, -9.5552e-01,  2.2955e-01,  1.8662e-04,  1.2429e-02])

Batched L2 normalization in PyTorch

python

batch_of_vectors = torch.rand((4, 64))
normalized_batch_of_vectors = F.normalize(batch_of_vectors, p=2.0, dim=1)
normalized_batch_of_vectors.shape, torch.norm(normalized_batch_of_vectors, dim=1) # all vectors will have length of 1.0

python

(torch.Size([4, 64]), tensor([1.0000, 1.0000, 1.0000, 1.0000]))

8. Linear layer + chunking trick (`torch.chunk`)

This is a creative trick I’ve found recently here. Let’s say you want to map your input into N different linear projections. You can do this by creating N nn.Linear layers and doing forward pass with all N of them OR you can create a single linear layer, do one forward pass and just chunk the output into N pieces. This method usually leads to higher performance, so it’s a nice trick to remember.

python

d = 1024
batch = torch.rand((8, d))
layers = nn.Linear(d, 128, bias=False), nn.Linear(d, 128, bias=False), nn.Linear(d, 128, bias=False)
one_layer = nn.Linear(d, 128 * 3, bias=False)

python

%%timeit
o1 = layers[0](batch)
o2 = layers[1](batch)
o3 = layers[2](batch)

289 µs ± 30.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

python

%%timeit
o1, o2, o3 = torch.chunk(one_layer(batch), 3, dim=1)

202 µs ± 8.09 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

9. Masked select (`torch.masked_select`)

Sometimes you will need to do your calculations only on some portion of the input tensors. To give you an example: you want to calculate loss only on the tensors that fulfuill some condition. In order to do so, just use torch.masked_select - note that this operation can be used when gradient is required too.

python

data = torch.rand((3, 3)).requires_grad_()
print(data)
mask = data > data.mean()
print(mask)
torch.masked_select(data, mask)

python

tensor([[0.0582, 0.7170, 0.7713],
        [0.9458, 0.2597, 0.6711],
        [0.2828, 0.2232, 0.1981]], requires_grad=True)
tensor([[False,  True,  True],
        [ True, False,  True],
        [False, False, False]])
tensor([0.7170, 0.7713, 0.9458, 0.6711], grad_fn=<MaskedSelectBackward>)

Applying mask on tensor directly

Similar behaviour can be achiveded by using mask as an “indexer” of the input tensor.

python

data[mask]

python

tensor([0.7170, 0.7713, 0.9458, 0.6711], grad_fn=<IndexBackward>)

Sometimes, a desirable solution is to fill all False values from mask with zeros, which can be done like this:

python

data * mask

python

tensor([[0.0000, 0.7170, 0.7713],
        [0.9458, 0.0000, 0.6711],
        [0.0000, 0.0000, 0.0000]], grad_fn=<MulBackward0>)

10. Conditional tensors with `torch.where`

This functions comes in handy when you want to combine two tensors with the condition - if it’s true, then take element from first tensor, if it’s false - take from the second one.
Docs: https://pytorch.org/docs/1.7.1/generated/torch.where.html#torch.where

python

x = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0], requires_grad=True)
y = -x
condition_or_mask = x <= 3.0
torch.where(condition_or_mask, x, y)

python

tensor([ 1.,  2.,  3., -4., -5.], grad_fn=<SWhereBackward>)

11. Filling tensors with values at given positions (`Tensor.scatter`)

Use case for this function is the following - you want to fill one tensor with values from another tensor at given indices. It’s simpler to understand on a 1D tensor, so I will show it first and then proceed to the more advanced example.

python

data = torch.tensor([1, 2, 3, 4, 5])
index = torch.tensor([0, 1])
values = torch.tensor([-1, -2, -3, -4, -5])
data.scatter(0, index, values)

python

tensor([-1, -2,  3,  4,  5])

The above example is simple, but now see what happens if you change index to index = torch.tensor([0, 1, 4])

python

data = torch.tensor([1, 2, 3, 4, 5])
index = torch.tensor([0, 1, 4])
values = torch.tensor([-1, -2, -3, -4, -5])
data.scatter(0, index, values)

python

tensor([-1, -2,  3,  4, -3])

It’s counter-intuitive why the last value is now -3, right? This is the clou of the PyTorch scatter function. The index variable says at which position from data tensor should the i-th value from the values tensor be placed. I hope the below plain python equivalent of this operation will shed more light:

python

data_orig = torch.tensor([1, 2, 3, 4, 5])
index = torch.tensor([0, 1, 4])
values = torch.tensor([-1, -2, -3, -4, -5])
scattered = data_orig.scatter(0, index, values)

data = data_orig.clone()
for idx_in_values, where_to_put_the_value in enumerate(index):
    what_value_to_put = values[idx_in_values]
    data[where_to_put_the_value] = what_value_to_put
data, scattered

python

(tensor([-1, -2,  3,  4, -3]), tensor([-1, -2,  3,  4, -3]))

PyTorch scatter example on 2D data

Always remember that shape of the index is related to the shape of the values and values within index correspond to the positions in data.

python

data = torch.zeros((4, 4)).float()
index = torch.tensor([
    [0, 1],
    [2, 3],
    [0, 3],
    [1, 2]
])
values = torch.arange(1, 9).float().view(4, 2)
values, data.scatter(1, index, values)

python

(tensor([[1., 2.],
        [3., 4.],
        [5., 6.],
        [7., 8.]]),
tensor([[1., 2., 0., 0.],
        [0., 0., 3., 4.],
        [5., 0., 0., 6.],
        [0., 7., 8., 0.]]))

12. Image interpolation within network (`F.interpolate`)

When I was learning PyTorch it surprised me that you can actually resize the image (or any intermediate tensor) within the forward pass and maintain the gradient flow. This method is especially useful when working with CNNs and GANs.

python

# image from https://commons.wikimedia.org/wiki/File:A_female_British_Shorthair_at_the_age_of_20_months.jpg
img = Image.open("./cat.jpg")
img

python

to_pil_image(
    F.interpolate(to_tensor(img).unsqueeze(0),  # batch of size 1
                  mode="bilinear", 
                  scale_factor=2.0, 
                  align_corners=False).squeeze(0) # remove batch dimension
)

See how the gradient flow is preserved:

python

F.interpolate(to_tensor(img).unsqueeze(0).requires_grad_(),
                  mode="bicubic", 
                  scale_factor=2.0, 
                  align_corners=False)

python

tensor([[[[0.9216, 0.9216, 0.9216,  ..., 0.8361, 0.8272, 0.8219],
    [0.9214, 0.9214, 0.9214,  ..., 0.8361, 0.8272, 0.8219],
    [0.9212, 0.9212, 0.9212,  ..., 0.8361, 0.8272, 0.8219],
    ...,
    [0.9098, 0.9098, 0.9098,  ..., 0.3592, 0.3486, 0.3421],
    [0.9098, 0.9098, 0.9098,  ..., 0.3566, 0.3463, 0.3400],
    [0.9098, 0.9098, 0.9098,  ..., 0.3550, 0.3449, 0.3387]],

    [[0.6627, 0.6627, 0.6627,  ..., 0.5380, 0.5292, 0.5238],
    [0.6626, 0.6626, 0.6626,  ..., 0.5380, 0.5292, 0.5238],
    [0.6623, 0.6623, 0.6623,  ..., 0.5380, 0.5292, 0.5238],
    ...,
    [0.6196, 0.6196, 0.6196,  ..., 0.3631, 0.3525, 0.3461],
    [0.6196, 0.6196, 0.6196,  ..., 0.3605, 0.3502, 0.3439],
    [0.6196, 0.6196, 0.6196,  ..., 0.3589, 0.3488, 0.3426]],

    [[0.4353, 0.4353, 0.4353,  ..., 0.1913, 0.1835, 0.1787],
    [0.4352, 0.4352, 0.4352,  ..., 0.1913, 0.1835, 0.1787],
    [0.4349, 0.4349, 0.4349,  ..., 0.1913, 0.1835, 0.1787],
    ...,
    [0.3333, 0.3333, 0.3333,  ..., 0.3827, 0.3721, 0.3657],
    [0.3333, 0.3333, 0.3333,  ..., 0.3801, 0.3698, 0.3635],
    [0.3333, 0.3333, 0.3333,  ..., 0.3785, 0.3684, 0.3622]]]],
grad_fn=<UpsampleBicubic2DBackward1>)

13. Making grid of images (`torchvision.utils.make_grid`)

There is no need to copy-paste the code using matplotlib or some external libraries in order to display a grid of images when working with PyTorch and torchvision. Just use torchvision.utils.make_grid!
Docs: https://pytorch.org/vision/0.8/utils.html

python

from torchvision.utils import make_grid
from torchvision.transforms.functional import to_tensor, to_pil_image
from PIL import Image
img = Image.open("./cat.jpg")
to_pil_image(
    make_grid(
        [to_tensor(i) for i in [img, img, img]],
         nrow=2, # number of images in single row
         padding=5 # "frame" size
     )
)

Summary

I hope my post will help you in your journey to master PyTorch. Please share if you like it and don’t hesitate to ask questions in comments!

Using ResNet for MNIST in PyTorch 1.7

Marcin Zabłocki — Sat, 30 Jan 2021 16:00:00 +0000

This short post is a refreshed version of my early-2019 post about adjusting ResNet architecture for use with well known MNIST dataset. The goal of this post is to provide refreshed overview on this process for the beginners. Treat is a tutorial how to train a MNIST digits classifier using PyTorch 1.7 and Torchvision.

TL;DR

Tutorial on how to train ResNet for MNIST using PyTorch, updated for 2021. Link to google colab at the bottom.

Use ResNet from torchvision

If you’re new to PyTorch, then you should know that there is an official PyTorch library dedicated for computer vision problems - Torchvision. It provides stable and well tested implementation of various network architectures, including: ResNets in many variants (ResNet-18, ResNet-34, ResNet-51 and so on), ShuffleNets, MobileNets as well as many other ResNet-related ones, that improve the original architecture e.g. ResNeXt.

Torchvision also weights for those networks that are trained on ImageNet dataset and are ready to use for transfer learning. In this tutorial, we will be training ResNet-18 from scratch, as digit images from MNIST dataset are really different from the images included in the ImageNet.

python

from torchvision.models import resnet18
from torch import nn
model = resnet18(num_classes=10) # MNIST has 10 classes

The overall model architecture looks like this:

python

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  # Skipped for brevity
  (layer1): ...
  (layer2): ...
  (layer3): ...
  (layer4): ...
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=512, out_features=10, bias=True)
)

In order to adapt this architecture for MNIST there is one more thing required - input layer needs to accept single channel instead of 3 (MNIST images are single-channel = grayscale, whereas ImageNet are 3-channels = RGB).

python

model.conv1 = Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

And it’s done! No more modifications are required.

Load the dataset

Besides the models, Torchvision also gives you access to a lot of well known computer vision datasets, including MNIST - it will automatically download the dataset for you and provide you with PyTorch’s Dataset abstraction that it’s ready to use.

python

train_ds = MNIST("mnist", train=True, download=True)
test_ds = MNIST("mnist", train=False, download=True)
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)
test_dl = DataLoader(test_ds, batch_size=64)

Do not write training loops in PyTorch

In the previous post, I’ve showed the training loop boiler plate for PyTorch. Back in early 2019 there were no well established libraries for trianing PyTorch models. Fortunately, in the early 2021 there are plenty. One of them is PyTorch Lightning. The minimal code is as follows:

python

import pytorch_lightning as pl
from pytorch_lightning.core.decorators import auto_move_data

class ResNetMNIST(pl.LightningModule):
  def __init__(self):
    super().__init__()
    # define model and loss
    self.model = resnet18(num_classes=10)
    self.model.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    self.loss = nn.CrossEntropyLoss()

  @auto_move_data # this decorator automatically handles moving your tensors to GPU if required
  def forward(self, x):
    return self.model(x)
  
  def training_step(self, batch, batch_no):
    # implement single training step
    x, y = batch
    logits = self(x)
    loss = self.loss(logits, y)
    return loss
  
  def configure_optimizers(self):
    # choose your optimizer
    return torch.optim.RMSprop(self.parameters(), lr=0.005)

python

model = ResNetMNIST()
trainer = pl.Trainer(
    gpus=1, # use one GPU
    max_epochs=1, # set number of epochs
    progress_bar_refresh_rate=20 # set to >= 20 if running in Google Colab
)
trainer.fit(model, train_dl)

bash

  | Name  | Type             | Params
-------------------------------------------
0 | model | ResNet           | 11.2 M
1 | loss  | CrossEntropyLoss | 0     
-------------------------------------------
Epoch 0: 100% 938/938 [00:21<00:00, 43.77it/s, loss=0.116]

And it’s done! Now, save the model so that it can be used later for inference.

python

trainer.save_checkpoint("resnet18_mnist.pt")

Inference with PyTorch Lightning

The infernce code is pretty straightforward. I’ve only added tqdm so that it reports the progress.

python

from tqdm.autonotebook import tqdm

def get_prediction(x, model: pl.LightningModule):
  model.freeze() # prepares model for predicting
  probabilities = torch.softmax(model(x), dim=1)
  predicted_class = torch.argmax(probabilities, dim=1)
  return predicted_class, probabilities

inference_model = ResNetMNIST.load_from_checkpoint("resnet18_mnist.pt", map_location="cuda")

Loop through any data…

python

true_y, pred_y = [], []
for batch in tqdm(iter(test_dl), total=len(test_dl)):
  x, y = batch
  true_y.extend(y)
  preds, probs = get_prediction(x, inference_model)
  pred_y.extend(preds.cpu())

…and get the final classification report:

python

from sklearn.metrics import classification_report
print(classification_report(true_y, pred_y, digits=3))

              precision    recall  f1-score   support

           0      0.963     0.996     0.979       980
           1      0.991     0.993     0.992      1135
           2      0.974     0.980     0.977      1032
           3      0.994     0.975     0.985      1010
           4      0.990     0.981     0.985       982
           5      0.974     0.976     0.975       892
           6      0.975     0.981     0.978       958
           7      0.966     0.984     0.975      1028
           8      0.977     0.963     0.970       974
           9      0.988     0.960     0.974      1009

    accuracy                          0.979     10000
   macro avg      0.979     0.979     0.979     10000
weighted avg      0.979     0.979     0.979     10000

Summary

I hope you find this post useful for your first steps with neural networks and PyTorch. Do not hesitate to comment and / or ask any questions.

Additional links & resources

Jupyter notebook for this post in Google Colab
GitHub repo with this notebook: https://github.com/marrrcin/pytorch-resnet-mnist

Stroke Width Transform (SWT) for Python 3 & docker

Marcin Zabłocki — Mon, 11 Jan 2021 22:00:00 +0000

Back in august of 2017 I needed to use SWT algorithm to detect images that contain text in any form. At that time, there was no efficient enough implementation for my case, so I’ve decided to create Python bindings for libccv’s implementation of SWT. That was over 3 years ago - the solution got deprecated over time, now it’s officialy Python 3 era, with Docker all over the place. As the previous post still recieves some traffic and my GitHub repo is active, I have decided to update my solution to support newest stable libccv, Python 3.9 and Docker.

TL;DR

Stroke Width Transform for Python 3 - how to use a fast SWT implementation from libccv in Python 3.9 and/or Docker.

Prerequisites

Option A - local build:

g++ compiler
Python 3.9 (it will probably work with 3.6+ too)
cmake >= 3.17
libjpeg-dev, libpng-dev, libatlas-base-dev, libblas-dev installed
basic knowledge of C++
pybind11

Option B - docker image:

docker

SWT bindings for Python 3.9

Here is the how:

Download and compile libccv in the form of shared library
Implement bindings for the libccv using pybind11
Compile the library and use it

I just want to note, that I’m no C++ developer and I haven’t used pybind11 before, so in my opinion there is no experience with this toolset required.

Compling libccv for use in C++

There is actually not much to be done. I’ve cloned the original libccv repository and changed one flag in the makefile to make it work.

bash

CFLAGS := -O3 -fPIC -ffast-math -Wall $(CFLAGS)

See https://github.com/marrrcin/ccv/commit/5d945a6a02283880b4be4370d4dd6dd229402820

I recommend to just clone my fork from https://github.com/marrrcin/ccv/tree/stable :

bash

git clone --single-branch --branch stable --depth 1 https://github.com/marrrcin/ccv.git /ccv

Before the compilation starts, the following packages need to be installed in the system (it should work both on macOS and Linux):

bash

libjpeg-dev
libpng-dev
libatlas-base-dev
libblas-dev

After the required libraries are installed, I’ve just run the following command from the ./lib directory:

bash

./configure && make libccv.a

The compliation is rather a lengthy process if all of the optimization flags are enabled (which they are, by default). Also, when I run this compliation on macOS I experienced peak memory usage of 7GB, so keep this in mind. If you want to trade off library speed for the decreased compilation time/resources usage, replace the -O3 to either -O2, -O1 or -O0 do decrease / disable the compiler’s optimizations.

Implementing pybind11 bindings for SWT in libccv

I’ve created new cmake-based C++ project for a shared library and I’ve copied libccv.a and ccv.h files from the ./lib directory (of cloned libccv repo) into new folder within my project - also called lib.

The whole bindings for SWT consists of barely 41 lines of C++ code which I explain below.

csharp

// Include libccv headers to use stroke width transform

extern "C" {
    #include "lib/ccv.h"
}

// Include pybind11 headers to create Python bindings
#include <pybind11/pybind11.h>

namespace py = pybind11;

// Run SWT on the in-memory image (e.g. PNG or JPEG) and return the detected rectangles
py::list swt(char *bytes, int array_length){
    py::list result; // Initialize plain-Python List
    ccv_enable_default_cache();
    ccv_dense_matrix_t* image = 0;
    ccv_enable_default_cache();
    
    // Read in-memory image
    ccv_read((void*)bytes, &image, CCV_IO_GRAY | CCV_IO_ANY_STREAM, array_length);
    if(image != 0) {
        ccv_array_t *words = ccv_swt_detect_words(image, ccv_swt_default_params);
        if (words) {
            int i;
            for (i = 0; i < words->rnum; i++) {
                ccv_rect_t *rect = (ccv_rect_t *) ccv_array_get(words, i);
                py::dict item; // Initialize plain-Python dictionary and fill it with values
                item["x"] = rect->x;
                item["y"] = rect->y;
                item["width"] = rect->width;
                item["height"] = rect->height;
                result.append(item);
            }
            ccv_array_free(words);
        }
        ccv_matrix_free(image);
    }

    return result;
}

// Define pybind11 module
PYBIND11_MODULE(swt_python3, m) {
    m.doc() = "Python 3 compatible SWT (Stroke-Width-Transform) binding to libccv by Marcin Zabłocki";
    m.def("swt", &swt, "Apply Stroke-Width-Transform on input byte stream");
}

One key difference when compared to my old implementation is that with pybind11 I could use well-known Python structures such as lists and dictionaries directly from C++ code, making the whole “from C++ to Python” data passing a piece of cake 🎉.

In order to compile the bindings into a Python-usable package there are a few additional steps required.

Project needs to have CMakeLists.txt file in order for cmake to know how to compile it. In this file all of the necessary linker and pybind11 configurations are set:

python

cmake_minimum_required(VERSION 3.17)
project(swt_python3)

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_FLAGS "-O3")
set(CMAKE_CXX_FLAGS_DEBUG "-g -O3")

find_library(CCV ccv lib)
find_package(JPEG REQUIRED)
find_package(PNG REQUIRED)
#find_package(PythonLibs) # Either uncomment this or specify `PYTHON_INCLUDE_DIRS` variable during build time
include_directories(${PYTHON_INCLUDE_DIRS})

add_subdirectory(pybind11)
pybind11_add_module(swt_python3)
target_sources(swt_python3 PRIVATE swt_python3.cpp)
target_link_libraries(swt_python3 PRIVATE ${CCV} ${JPEG_LIBRARIES} ${PNG_LIBRARIES})

Project needs to know where to find pybind11. It can be specified as another CMakeLists.txt file in the pybind11 folder within the project:

python

include(FetchContent)

FetchContent_Declare(
        pybind11
        URL   "https://github.com/pybind/pybind11/archive/v2.6.1.tar.gz"
)

FetchContent_MakeAvailable(pybind11)

That’s it. I compile the wrapper using the following command:

bash

cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_INCLUDE_DIRS=/usr/local/include/python3.9 . && make

It creates a file called swt_python3.cpython-39m-darwin.so (on macOS) or swt_python3.cpython-39-x86_64-linux-gnu.so (on Linux) which is Python-ready module to import!

TIP: in order to find PYTHON_INCLUDE_DIRS just run:

bash

python -c "from sysconfig import get_paths as gp; print(gp()['include'])"

Using stroke width transform in Python 3

Compiled binding is ready to be imported directly from Python code, so if you want to use it anywhere, just copy .so file into your project’s directory and call:

python

from swt_python3 import swt

Invoking SWT on any image is as simple as:

python

buffer = open("input.jpg", "rb").read()
swt_result: List[dict] = swt(buffer, len(buffer))
for item in swt_result:
    x, y, width, height = [item[key] for key in ("x", "y", "width", "height")]
    print(x, y, width, height)

Thanks to the pybind11 magic, the interface is much more plesant than last time.

Input

Output

SWT for Python 3 in Docker

As many modern deployments are Docker-based, I’ve decided to also include pre-built docker image which includes compiled and ready to use SWT wrapper with Python 3.9. The image includes an example.py script with the code that runs SWT on an image and saves the result into another one.

bash

docker pull marrrcin/swt-python3-ccv:20210111
docker run --rm -it --entrypoint bash marrrcin/swt-python3-ccv:20210111

Dockerfile: https://github.com/marrrcin/swt-python/blob/86a7553d1861f4dd7114c01b6ae512090d7ee14f/Dockerfile

Summary

I hope you liked it, please do not hesitate to create an issue or PR on GitHub if you find any issues.

Additional links & resources

full project on my github: https://github.com/marrrcin/swt-python
Docker Hub with the image: https://hub.docker.com/r/marrrcin/swt-python3-ccv
original paper about SWT: https://www.microsoft.com/en-us/research/publication/stroke-width-transform/

Understanding & implementing SimCLR in PyTorch - an ELI5 guide

Marcin Zabłocki — Mon, 27 Jul 2020 14:57:48 +0200

Transfer learning and pre-training schemas for both NLP and Computer Vision have gained a lot of attention in the last months. Research showed that carefully designed unsupervised/self-supervised training can produce high quality base models and embeddings that greatly decrease the amount of data needed to obtain good classification models downstream. This approach becomes more and more important as the companies collect a lot of data from which only a fraction can be labelled by humans - either due to large cost of labelling process or due to some time constraints.

Here I explore SimCLR pre-training framework proposed by Google in this arxiv paper. I will explain the SimCLR and its contrastive loss function step by step, starting from naive implementation in PyTorch, followed by faster, vectorized one. Then I will show how to use SimCLR’s pre-training routine to first build image embeddings using EfficientNet network architecture and finally I will show how to build a classifier on top of it.

TL;DR

This post covers:

understanding the SimCLR framework with code samples in PyTorch
from scratch explanation & implementation of SimCLR’s loss function (NT-Xent) in PyTorch
pre-training image embeddings using EfficientNet architecture
training classifier by using transfer learning from the pre-trained embeddings

Understanding SimCLR framework

In general, SimCLR is a simple framework for contrastive learning of visual representations. It’s not any new framework for deep learning, it’s a set of fixed steps that one should follow in order to train image embeddings of good quality. I drew a schema which explains the flow and the whole representation learning process (click to zoom).

The flow is as follows (from left to right):

Take an input image
Prepare 2 random augmentations on the image, including: rotations, hue/saturation/brightness changes, zooming, cropping, etc. The range of augmentations as well as analysis which ones work best are discussed in detail in the paper.
Run a deep neural network (preferably convolutional one, like ResNet50) to obtain image representations (embeddings) for those augmented images.
Run a small fully connected linear neural network to project embeddings into another vector space.
Calculate contrastive loss and run backpropagation through both networks. Contrastive loss decreases when projections coming from the same image are similar. The similarity between projections can be arbitrary, here I will use cosine similarity, same as in the paper.

Contrastive loss function

Theory behind contrastive loss function

One can reason about contrastive loss function form two angles:

Contrastive loss decreases when projections of augmented images coming from the same input image are similar.
For two augmented images: (i), (j) (coming from the same input image - I will call them “positive” pair later on), the contrastive loss for (i) tries to identify (j) among other images (“negative” examples) that are in the same batch .

Formal definition of the loss for pair of positive examples (i) and (j) defined as:

The final loss is an arithmetic mean of the losses for all positive pairs in the batch:

(keep in mind that the indexing in l(2k-1, 2k) + l(2k, 2k-1) is purely dependent on how you implement the loss - I find it easier to understand when I reason about them as l(i,j) + l(j,i)).

Contrastive loss function - implementation in PyTorch, ELI5 version

It’s much easier to implement the loss function without vectorization first and then follow up with the vectorization phase.

python

import torch
from torch import nn
import torch.nn.functional as F

python

class ContrastiveLossELI5(nn.Module):
    def __init__(self, batch_size, temperature=0.5, verbose=True):
        super().__init__()
        self.batch_size = batch_size
        self.register_buffer("temperature", torch.tensor(temperature))
        self.verbose = verbose
            
    def forward(self, emb_i, emb_j):
        """
        emb_i and emb_j are batches of embeddings, where corresponding indices are pairs
        z_i, z_j as per SimCLR paper
        """
        z_i = F.normalize(emb_i, dim=1)
        z_j = F.normalize(emb_j, dim=1)

        representations = torch.cat([z_i, z_j], dim=0)
        similarity_matrix = F.cosine_similarity(representations.unsqueeze(1), representations.unsqueeze(0), dim=2)
        if self.verbose: print("Similarity matrix\n", similarity_matrix, "\n")
            
        def l_ij(i, j):
            z_i_, z_j_ = representations[i], representations[j]
            sim_i_j = similarity_matrix[i, j]
            if self.verbose: print(f"sim({i}, {j})={sim_i_j}")
                
            numerator = torch.exp(sim_i_j / self.temperature)
            one_for_not_i = torch.ones((2 * self.batch_size, )).to(emb_i.device).scatter_(0, torch.tensor([i]), 0.0)
            if self.verbose: print(f"1{{k!={i}}}",one_for_not_i)
            
            denominator = torch.sum(
                one_for_not_i * torch.exp(similarity_matrix[i, :] / self.temperature)
            )    
            if self.verbose: print("Denominator", denominator)
                
            loss_ij = -torch.log(numerator / denominator)
            if self.verbose: print(f"loss({i},{j})={loss_ij}\n")
                
            return loss_ij.squeeze(0)

        N = self.batch_size
        loss = 0.0
        for k in range(0, N):
            loss += l_ij(k, k + N) + l_ij(k + N, k)
        return 1.0 / (2*N) * loss

Explanation

Contrastive loss needs to know the batch size and temperature (scaling) parameter. You can find details about setting the optimal temperature parameter in the paper.

My implementation of the forward of the contrastive loss takes two parameters. First one will be a batch projection of images after first augmentation, the second will be a batch projection of images after second augmentation.

Projections needs to be normalized first, hence:

python

z_i = F.normalize(emb_i, dim=1)
z_j = F.normalize(emb_j, dim=1)

All representations are concatenated together in order to efficiently calculate cosine similarities between each image pair.

python

representations = torch.cat([z_i, z_j], dim=0)
similarity_matrix = F.cosine_similarity(representations.unsqueeze(1), representations.unsqueeze(0), dim=2)

Next is the naive implementation of l(i,j) for clarity and easiness of understanding. The code bellow almost directly implements the equation:

python

def l_ij(i, j):
    z_i_, z_j_ = representations[i], representations[j]
    sim_i_j = similarity_matrix[i, j]
    numerator = torch.exp(sim_i_j / self.temperature)
    one_for_not_i = torch.ones((2 * self.batch_size, )).to(emb_i.device).scatter_(0, torch.tensor([i]), 0.0)
    denominator = torch.sum(
        one_for_not_i * torch.exp(similarity_matrix[i, :] / self.temperature)
    )

    loss_ij = -torch.log(numerator / denominator)
    return loss_ij.squeeze(0)

Then, the final loss for the batch is computed as an arithmetic mean of all combinations of positive examples:

python

N = self.batch_size
loss = 0.0
for k in range(0, N):
    loss += l_ij(k, k + N) + l_ij(k + N, k)
return 1.0 / (2*N) * loss

Now, let’s run it with verbose mode to see what’s inside.

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]])
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]])
loss_eli5 = ContrastiveLossELI5(batch_size=3, temperature=1.0, verbose=True)
loss_eli5(I, J)

bash

Similarity matrix
    tensor([[ 1.0000, -0.1240,  0.9648,  0.8944, -0.0948,  0.9679],
        [-0.1240,  1.0000, -0.3807,  0.3328,  0.9996, -0.3694],
        [ 0.9648, -0.3807,  1.0000,  0.7452, -0.3534,  0.9999],
        [ 0.8944,  0.3328,  0.7452,  1.0000,  0.3604,  0.7533],
        [-0.0948,  0.9996, -0.3534,  0.3604,  1.0000, -0.3419],
        [ 0.9679, -0.3694,  0.9999,  0.7533, -0.3419,  1.0000]]) 

sim(0, 3)=0.8944272398948669
1{k!=0} tensor([0., 1., 1., 1., 1., 1.])
Denominator tensor(9.4954)
loss(0,3)=1.3563847541809082

sim(3, 0)=0.8944272398948669
1{k!=3} tensor([1., 1., 1., 0., 1., 1.])
Denominator tensor(9.5058)
loss(3,0)=1.357473373413086

sim(1, 4)=0.9995677471160889
1{k!=1} tensor([1., 0., 1., 1., 1., 1.])
Denominator tensor(6.3699)
loss(1,4)=0.8520082831382751

sim(4, 1)=0.9995677471160889
1{k!=4} tensor([1., 1., 1., 1., 0., 1.])
Denominator tensor(6.4733)
loss(4,1)=0.8681114912033081

sim(2, 5)=0.9999250769615173
1{k!=2} tensor([1., 1., 0., 1., 1., 1.])
Denominator tensor(8.8348)
loss(2,5)=1.1787779331207275

sim(5, 2)=0.9999250769615173
1{k!=5} tensor([1., 1., 1., 1., 1., 0.])
Denominator tensor(8.8762)
loss(5,2)=1.1834462881088257

tensor(1.1327)

A few things happened there, but by going back and forward between the verbose logs and the equation, everything should become clear. The indexing jumps by batch size (first l(0,3), l(3,0) then l(1,4), l(4,1) because of the way the similarity matrix was constructed. First row of the similarity_matrix is:

python

[ 1.0000, -0.1240,  0.9648,  0.8944, -0.0948,  0.9679]

Remember the input:

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]])
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]])

Now:

1.0000 is the cosine similarity between I[0] and I[0] ([1.0, 2.0] and [1.0, 2.0])

-0.1240 is the cosine similarity between I[0] and I[1] ([1.0, 2.0] and [3.0, -2.0])

-0.0948 is the cosine similarity between I[0] and J[2] ([1.0, 2.0] and [2.8, -1.75])

… and so on.

Let’s see if the loss decreases if the similarity between first image projection increases:

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]])
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]])
J = torch.tensor([[1.0, 1.75], [2.8, -1.75], [1.0, 4.7]]) # note the change
ContrastiveLossELI5(3, 1.0, verbose=False)(I, J)

bash

tensor(1.0996)

Indeed the loss decreases! Now I will follow up with the vectorized implementation.

Contrastive loss function - implementation in PyTorch, vectorized version

Performance of naive implementation is really poor (mostly due to manual loop), see the results:

python

contrastive_loss_eli5 = ContrastiveLossELI5(3, 1.0, verbose=False)
I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]], requires_grad=True)
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]], requires_grad=True)

python

%%timeit
contrastive_loss_eli5(I, J)
838 µs ± 23.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Once I understood the internals of the loss, it’s easy to vectorize it and remove the manual loop:

python

class ContrastiveLoss(nn.Module):
    def __init__(self, batch_size, temperature=0.5):
        super().__init__()
        self.batch_size = batch_size
        self.register_buffer("temperature", torch.tensor(temperature))
        self.register_buffer("negatives_mask", (~torch.eye(batch_size * 2, batch_size * 2, dtype=bool)).float())
            
    def forward(self, emb_i, emb_j):
        """
        emb_i and emb_j are batches of embeddings, where corresponding indices are pairs
        z_i, z_j as per SimCLR paper
        """
        z_i = F.normalize(emb_i, dim=1)
        z_j = F.normalize(emb_j, dim=1)

        representations = torch.cat([z_i, z_j], dim=0)
        similarity_matrix = F.cosine_similarity(representations.unsqueeze(1), representations.unsqueeze(0), dim=2)
        
        sim_ij = torch.diag(similarity_matrix, self.batch_size)
        sim_ji = torch.diag(similarity_matrix, -self.batch_size)
        positives = torch.cat([sim_ij, sim_ji], dim=0)
        
        nominator = torch.exp(positives / self.temperature)
        denominator = self.negatives_mask * torch.exp(similarity_matrix / self.temperature)
    
        loss_partial = -torch.log(nominator / torch.sum(denominator, dim=1))
        loss = torch.sum(loss_partial) / (2 * self.batch_size)
        return loss

python

contrastive_loss = ContrastiveLoss(3, 1.0)
contrastive_loss(I, J).item() - contrastive_loss_eli5(I, J).item()
0.0

The difference should be zero or close to zero (< 1e-6 due to fp arithmetics). Performance comparison:

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]], requires_grad=True)
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]], requires_grad=True)

python

%%timeit
contrastive_loss_eli5(I, J)
918 µs ± 60.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

python

%%timeit
contrastive_loss(I, J)
272 µs ± 9.18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Almost 4x improvement, it works.

Pre-training image embeddings using SimCLR with EfficientNet

Once the loss function is established and understood it’s time to make a good use of it. I will pre-train image embeddings using EfficientNet architecture, following the SimCLR framework. For convenience, I’ve implemented a few utility functions and classes that I will explain briefly below. The training code is structured using PyTorch-Lightning.

I will use a great EfficientNet [https://arxiv.org/abs/1905.11946] implementation by Luke Melas-Kyriazi from GitHub, already pre-trained on ImageNet (transfer learning inception!). The dataset I choose is STL10 (from torchvision) as it contains both training and unlabeled splits for unsupervised / self-supervised learning tasks.

My goal here is to demonstrate the whole SimCLR flow from start to the end. I had no intent to reach new SOTA with the presented configuration.

Utility functions for image augmentations

Training with SimCLR produces good image embeddings that are not affected by image transformations - this is because during training, various data augmentations are done to force the network to understand the contents of the images regardless of i.e color of the image or the position where the object in the image is placed. SimCLR’s authors say that composition of data augmentations plays a critical role in defining effective predictive tasks and also contrastive learning needs stronger data augmentation than supervised learning. To sum this up: when pre-training the image embeddings it’s good to make this task difficult for the network to learn by strongly augmenting the images in order to generalize better afterwards.

I strongly advise to read the SimCLR’s paper and it’s appendix as they did ablation studies on which data augmentations bring the best effects on the embeddings.

To keep this blog-post simple to go through, I will mostly use built-in Torchvision’s data augmentations , with one additional one - random resized rotation.

python

def random_rotate(image):
    if random.random() > 0.5:
        return tvf.rotate(image, angle=random.choice((0, 90, 180, 270)))
    return image

python

class ResizedRotation():
    def __init__(self, angle, output_size=(96, 96)):
        self.angle = angle
        self.output_size = output_size
        
    def angle_to_rad(self, ang): return np.pi * ang / 180.0
        
    def __call__(self, image):
        w, h = image.size
        new_h = int(np.abs(w * np.sin(self.angle_to_rad(90 - self.angle))) + np.abs(h * np.sin(self.angle_to_rad(self.angle))))
        new_w = int(np.abs(h * np.sin(self.angle_to_rad(90 - self.angle))) + np.abs(w * np.sin(self.angle_to_rad(self.angle))))
        img = tvf.resize(image, (new_w, new_h))
        img = tvf.rotate(img, self.angle)
        img = tvf.center_crop(img, self.output_size)
        return img

python

class WrapWithRandomParams():
    def __init__(self, constructor, ranges):
        self.constructor = constructor
        self.ranges = ranges
    
    def __call__(self, image):
        randoms = [float(np.random.uniform(low, high)) for _, (low, high) in zip(range(len(self.ranges)), self.ranges)]
        return self.constructor(*randoms)(image)

python

from torchvision.datasets import STL10
import torchvision.transforms.functional as tvf
from torchvision import transforms
import numpy as np

A brief look on the transformation results:

python

stl10_unlabeled = STL10(".", split="unlabeled", download=True)

python

idx = 123

python

random_resized_rotation = WrapWithRandomParams(lambda angle: ResizedRotation(angle), [(0.0, 360.0)])

python

random_resized_rotation(tvf.resize(stl10_unlabeled[idx][0], (96, 96)))

Automatic data augmentation wrapper

Here I’ve also implemented a utility dataset wrapper that automatically applies random data augmentations every time when the image is retrieved. It can be easily used with any image dataset as long as it follows the simple interface of returning tuple with (PIL Image, anything). This wrapper can be set to return a deterministic transformation with the debug flag set to True. Note that there is a preprocess step that applies ImageNet-originated data standardization as I’m using already pre-trained EfficientNet.

python

from torch.utils.data import Dataset, DataLoader, SubsetRandomSampler, SequentialSampler
import random

python

class PretrainingDatasetWrapper(Dataset):
    def __init__(self, ds: Dataset, target_size=(96, 96), debug=False):
        super().__init__()
        self.ds = ds
        self.debug = debug
        self.target_size = target_size
        if debug:
            print("DATASET IN DEBUG MODE")
        
        # I will be using network pre-trained on ImageNet first, which uses this normalization.
        # Remove this, if you're training from scratch or apply different transformations accordingly
        self.preprocess = transforms.Compose([
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ])
        
        random_resized_rotation = WrapWithRandomParams(lambda angle: ResizedRotation(angle, target_size), [(0.0, 360.0)])
        self.randomize = transforms.Compose([
            transforms.RandomResizedCrop(target_size, scale=(1/3, 1.0), ratio=(0.3, 2.0)),
            transforms.RandomChoice([
                transforms.RandomHorizontalFlip(p=0.5),
                transforms.Lambda(random_rotate)
            ]),
            transforms.RandomApply([
                random_resized_rotation
            ], p=0.33),
            transforms.RandomApply([
                transforms.ColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0.2)
            ], p=0.8),
            transforms.RandomGrayscale(p=0.2)
        ])
    
    def __len__(self): return len(self.ds)
    
    def __getitem_internal__(self, idx, preprocess=True):
        this_image_raw, _ = self.ds[idx]
        
        if self.debug:
            random.seed(idx)
            t1 = self.randomize(this_image_raw)
            random.seed(idx + 1)
            t2 = self.randomize(this_image_raw)
        else:
            t1 = self.randomize(this_image_raw)
            t2 = self.randomize(this_image_raw)
        
        if preprocess:
            t1 = self.preprocess(t1)
            t2 = self.preprocess(t2)
        else:
            t1 = transforms.ToTensor()(t1)
            t2 = transforms.ToTensor()(t2)

        return (t1, t2), torch.tensor(0)

    def __getitem__(self, idx):
        return self.__getitem_internal__(idx, True)
    
    def raw(self, idx):
        return self.__getitem_internal__(idx, False)

python

ds = PretrainingDatasetWrapper(stl10_unlabeled, debug=False)

python

tvf.to_pil_image(ds[idx][0][0])

python

tvf.to_pil_image(ds[idx][0][1])

python

tvf.to_pil_image(ds.raw(idx)[0][1])

python

tvf.to_pil_image(ds.raw(idx)[0][0])

SimCLR neural network for embeddings

Here I define the ImageEmbedding neural network which is based on EfficientNet-b0 architecture. I swap out the last layer of pre-trained EfficientNet with identity function and add projection for image embeddings on top of it (following the SimCLR paper) with Linear-ReLU-Linear layers. It was shown in the paper that the non-linear projection head (i.e Linear-ReLU-Linear) improves the quality of the embeddings.

python

from efficientnet_pytorch import EfficientNet

python

class ImageEmbedding(nn.Module):       
    class Identity(nn.Module):
        def __init__(self): super().__init__()

        def forward(self, x):
            return x
    
        
    def __init__(self, embedding_size=1024):
        super().__init__()
        
        base_model = EfficientNet.from_pretrained("efficientnet-b0")
        internal_embedding_size = base_model._fc.in_features
        base_model._fc = ImageEmbedding.Identity()
        
        self.embedding = base_model
        
        self.projection = nn.Sequential(
            nn.Linear(in_features=internal_embedding_size, out_features=embedding_size),
            nn.ReLU(),
            nn.Linear(in_features=embedding_size, out_features=embedding_size)
        )

    def calculate_embedding(self, image):
        return self.embedding(image)

    def forward(self, X):
        image = X
        embedding = self.calculate_embedding(image)
        projection = self.projection(embedding)
        return embedding, projection

Next is the implementation of PyTorch-Lightning-based training module that orchestrates everything together:

hyper-parameters handling
SimCLR ImageEmbedding network
STL10 dataset
optimizer
forward step

As the PretrainingDatasetWrapper I’ve implemented returns tuple of: (Image1, Image2), dummy class, the forward step for this module is straightforward - it needs to produce two batches of embeddings and calculate contrastive loss function:

python

(X, Y), y = batch
embX, projectionX = self.forward(X)
embY, projectionY = self.forward(Y)
loss = self.loss(projectionX, projectionY)

python

from torch.multiprocessing import cpu_count
from torch.optim import RMSprop
import pytorch_lightning as pl

python

class ImageEmbeddingModule(pl.LightningModule):
    def __init__(self, hparams):
        hparams = Namespace(**hparams) if isinstance(hparams, dict) else hparams
        super().__init__()
        self.hparams = hparams
        self.model = ImageEmbedding()
        self.loss = ContrastiveLoss(hparams.batch_size)
    
    def total_steps(self):
        return len(self.train_dataloader()) // self.hparams.epochs
    
    def train_dataloader(self):
        return DataLoader(PretrainingDatasetWrapper(stl10_unlabeled, 
                                             debug=getattr(self.hparams, "debug", False)),
                          batch_size=self.hparams.batch_size, 
                          num_workers=cpu_count(),
                          sampler=SubsetRandomSampler(list(range(hparams.train_size))),
                         drop_last=True)
    
    def val_dataloader(self):
        return DataLoader(PretrainingDatasetWrapper(stl10_unlabeled,
                                            debug=getattr(self.hparams, "debug", False)),
                          batch_size=self.hparams.batch_size, 
                          shuffle=False,
                          num_workers=cpu_count(),
                          sampler=SequentialSampler(list(range(hparams.train_size + 1, hparams.train_size + hparams.validation_size))),
                         drop_last=True)
    
    def forward(self, X):
        return self.model(X)
    
    def step(self, batch, step_name = "train"):
        (X, Y), y = batch
        embX, projectionX = self.forward(X)
        embY, projectionY = self.forward(Y)
        loss = self.loss(projectionX, projectionY)
        loss_key = f"{step_name}_loss"
        tensorboard_logs = {loss_key: loss}

        return { ("loss" if step_name == "train" else loss_key): loss, 'log': tensorboard_logs,
                        "progress_bar": {loss_key: loss}}
    
    def training_step(self, batch, batch_idx):
        return self.step(batch, "train")
    
    def validation_step(self, batch, batch_idx):
        return self.step(batch, "val")
    
    def validation_end(self, outputs):
        if len(outputs) == 0:
            return {"val_loss": torch.tensor(0)}
        else:
            loss = torch.stack([x["val_loss"] for x in outputs]).mean()
            return {"val_loss": loss, "log": {"val_loss": loss}}

    def configure_optimizers(self):
        optimizer = RMSprop(self.model.parameters(), lr=self.hparams.lr)
        return [optimizer], []

Initial hyper-parameters. Batch size of 128 works fine with EfficientNet-B0 on GTX1070. Note that I’ve limited training dataset to first 10k images from STL10 for convenience of running this blogpost in the form of Jupyter Notebook / Google Colab.

Important! SimCLR greatly benefits from large batch sizes - it should be set to as high value as possible given the GPU / cluster limits.

python

from argparse import Namespace
hparams = Namespace(
    lr=1e-3,
    epochs=50,
    batch_size=160,
    train_size=10000,
    validation_size=1000
)

Finding good initial-learning rate using LRFinder algorithm

I use pytorch-lightning’s built-in LRFinder algorithm to find initial learning rate.

python

module = ImageEmbeddingModule(hparams)
t = pl.Trainer(gpus=1)
lr_finder = t.lr_find(module)
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0]

   | Name  | Type            | Params
------------------------------------------
0 | model | ImageEmbedding  | 6 M  
1 | loss  | ContrastiveLoss | 0

python

lr_finder.plot(show=False, suggest=True)
lr_finder.suggestion()
0.000630957344480193

I also use W&B Logging to keep track of my experiments:

python

from pytorch_lightning.loggers import WandbLogger

python

hparams = Namespace(
    lr=0.000630957344480193,
    epochs=10,
    batch_size=160,
    train_size=20000,
    validation_size=1000
)
module = ImageEmbeddingModule(hparams)
logger = WandbLogger(project="simclr-blogpost")
logger.watch(module, log="all", log_freq=50)
trainer = pl.Trainer(gpus=1, logger=logger)

python

trainer.fit(module)
| Name  | Type            | Params
------------------------------------------
0 | model | ImageEmbedding  | 6 M   
1 | loss  | ContrastiveLoss | 0

After the training completes, embeddings are ready for usage for the downstream tasks.

Image classifier on top of SimCLR embeddings

Once the embeddings are trained, they can be used to train the classifier on top of them - either by fine tuning the whole network or by freezing the base network with embeddings and learning linear classifiers on top of it - I show the latter below.

Save the weights of neural network with embeddings

I save the whole network in a form on a checkpoint. Only the internal part of the network will be used later with the classifier (projection layers will be discarded).

python

checkpoint_file = "efficientnet-b0-stl10-embeddings.ckpt"

python

trainer.save_checkpoint(checkpoint_file)

python

trainer.logger.experiment.log_artifact(checkpoint_file, type="model")

Classifier module

Again, I define a custom module - this time it uses already existing embeddings and freezes the base model weights on demand. Note the SimCLRClassifier.embeddings are only the EfficientNet part of the whole network used before - the projection head is discarded.

python

class SimCLRClassifier(nn.Module):
    def __init__(self, n_classes, freeze_base, embeddings_model_path, hidden_size=512):
        super().__init__()
        
        base_model = ImageEmbeddingModule.load_from_checkpoint(embeddings_model_path).model
        
        self.embeddings = base_model.embedding
        
        if freeze_base:
            print("Freezing embeddings")
            for param in self.embeddings.parameters():
                param.requires_grad = False
                
        # Only linear projection on top of the embeddings should be enough
        self.classifier = nn.Linear(in_features=base_model.projection[0].in_features, 
                      out_features=n_classes if n_classes > 2 else 1)

    
    def forward(self, X, *args):
        emb = self.embeddings(X)
        return self.classifier(emb)

Classifier training code

Classifier training code again uses PyTorch lightning, so I’m skipping in-depth explanation.

python

from torch import nn
from torch.optim.lr_scheduler import CosineAnnealingLR

python

class SimCLRClassifierModule(pl.LightningModule):
    def __init__(self, hparams):
        super().__init__()
        hparams = Namespace(**hparams) if isinstance(hparams, dict) else hparams
        self.hparams = hparams
        self.model = SimCLRClassifier(hparams.n_classes, hparams.freeze_base, 
                                      hparams.embeddings_path,
                                      self.hparams.hidden_size)
        self.loss = nn.CrossEntropyLoss()
    
    def total_steps(self):
        return len(self.train_dataloader()) // self.hparams.epochs
    
    def preprocessing(seff):
        return transforms.Compose([
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ])
    
    def get_dataloader(self, split):
        return DataLoader(STL10(".", split=split, transform=self.preprocessing()),
                          batch_size=self.hparams.batch_size, 
                          shuffle=split=="train",
                          num_workers=cpu_count(),
                         drop_last=False)
    
    def train_dataloader(self):
        return self.get_dataloader("train")
    
    def val_dataloader(self):
        return self.get_dataloader("test")
    
    def forward(self, X):
        return self.model(X)
    
    def step(self, batch, step_name = "train"):
        X, y = batch
        y_out = self.forward(X)
        loss = self.loss(y_out, y)
        loss_key = f"{step_name}_loss"
        tensorboard_logs = {loss_key: loss}

        return { ("loss" if step_name == "train" else loss_key): loss, 'log': tensorboard_logs,
                        "progress_bar": {loss_key: loss}}
    
    def training_step(self, batch, batch_idx):
        return self.step(batch, "train")
    
    def validation_step(self, batch, batch_idx):
        return self.step(batch, "val")
    
    def test_step(self, batch, batch_idx):
        return self.step(Batch, "test")
    
    def validation_end(self, outputs):
        if len(outputs) == 0:
            return {"val_loss": torch.tensor(0)}
        else:
            loss = torch.stack([x["val_loss"] for x in outputs]).mean()
            return {"val_loss": loss, "log": {"val_loss": loss}}

    def configure_optimizers(self):
        optimizer = RMSprop(self.model.parameters(), lr=self.hparams.lr)
        schedulers = [
            CosineAnnealingLR(optimizer, self.hparams.epochs)
        ] if self.hparams.epochs > 1 else []
        return [optimizer], schedulers

It’s worth mentioning here that training with a frozen base model gives a great performance boost during training as the gradients need to be calculated only for a single layer. Additionally, by utilizing good embeddings, only a few epochs are required to reach a good quality classifier with single linear projection.

python

hparams_cls = Namespace(
    lr=1e-3,
    epochs=5,
    batch_size=160,
    n_classes=10,
    freeze_base=True,
    embeddings_path="./efficientnet-b0-stl10-embeddings.ckpt",
    hidden_size=512
)
module = SimCLRClassifierModule(hparams_cls)
logger = WandbLogger(project="simclr-blogpost-classifier")
logger.watch(module, log="all", log_freq=10)
trainer = pl.Trainer(gpus=1, max_epochs=hparams_cls.epochs, logger=logger)

python

lr_find_cls = trainer.lr_find(module)
| Name  | Type             | Params
-------------------------------------------
0 | model | SimCLRClassifier | 4 M   
1 | loss  | CrossEntropyLoss | 0  
LR finder stopped early due to diverging loss.

python

lr_find_cls.plot(show=False, suggest=True)
lr_find_cls.suggestion()
0.003981071705534969

python

hparams_cls = Namespace(
    lr=0.003981071705534969,
    epochs=5,
    batch_size=160,
    n_classes=10,
    freeze_base=True,
    embeddings_path="./efficientnet-b0-stl10-embeddings.ckpt",
    hidden_size=512
)
module = SimCLRClassifierModule(hparams_cls)

python

trainer.fit(module)
| Name  | Type             | Params
-------------------------------------------
0 | model | SimCLRClassifier | 4 M   
1 | loss  | CrossEntropyLoss | 0

Evaluation

Here I define a utility function for evaluating the model using the provided data loader. Note the transfer between GPU and CPU and storing all results in-memory might be not effective for large datasets!

python

from sklearn.metrics import classification_report
def evaluate(data_loader, module):
    with torch.no_grad():
        progress = ["/", "-", "\\", "|", "/", "-", "\\", "|"]
        module.eval().cuda()
        true_y, pred_y = [], []
        for i, batch_ in enumerate(data_loader):
            X, y = batch_
            print(progress[i % len(progress)], end="\r")
            y_pred = torch.argmax(module(X.cuda()), dim=1)
            true_y.extend(y.cpu())
            pred_y.extend(y_pred.cpu())
        print(classification_report(true_y, pred_y, digits=3))
        return true_y, pred_y

python

_ = evaluate(module.val_dataloader(), module)

bash

                  precision    recall  f1-score   support
    
               0      0.856     0.864     0.860       800
               1      0.714     0.701     0.707       800
               2      0.903     0.919     0.911       800
               3      0.678     0.599     0.636       800
               4      0.665     0.746     0.703       800
               5      0.633     0.564     0.597       800
               6      0.729     0.781     0.754       800
               7      0.678     0.709     0.693       800
               8      0.868     0.910     0.888       800
               9      0.862     0.801     0.830       800
    
        accuracy                          0.759      8000
       macro avg      0.759     0.759     0.758      8000
    weighted avg      0.759     0.759     0.758      8000

Summary

I hope you find my explanation of the SimCLR framework useful. Feel free to ask anything, comment, play with the notebook and share this post with the ML community!

Additional links and resources

SimCLR paper - arxiv.org/pdf/2002.05709.pdf
EfficientNet paper - arxiv.org/abs/1905.11946
this notebook on GitHub - link
this notebook on Google Colab - link

Custom classifier on top of BERT-like Language Model - guide

Marcin Zabłocki — Mon, 23 Mar 2020 23:00:00 +0000

Last time I wrote about training the language models from scratch, you can find this post here. Now it’s time to take your pre-trained lamnguage model at put it into good use by fine-tuning it for real world problem, i.e text classification or sentiment analysis. In this post I will show how to take pre-trained language model and build custom classifier on top of it. As in the previous post - I cover all of the important parts in details and I point out possible mistakes.

Why custom model? There are two answers. First is that the fun in deep learning begins only when you can do something custom with your model. Experiments, experiments and more experiments! Second thing is that by implmenting some parts on your own, you gain better understaing of different parts of the modeling itself, but also the whole training/fine-tuning process.

TL;DR

This post covers:

taking existing pre-trained language model and understanding it’s output - here I use PolBERTa trained for Polish language
building custom classification head on top of the LM
using fast tokenizers to efficiently tokenize and pad input text as well as prepare attention masks
preparing reproducible training code with PyTorch Lightning
finding good starting learning rate for the model
validating the trained model on PolEmo 2.0 dataset (benchmark for Polish language sentiment analysis with 4 classes)

The whole post is available as a Colab Notebook, linked below in Additional links and resources section.

Libraries and preparation

In this post I will use pretty standard set of ML-related libraries, including: transformers & tokenizers from HuggingFace, PyTorch Lightning, pandas, scikit-learn and LR Finder.

python

!pip install transformers tokenizers pytorch-lightning

After running the cell below, you will probably have to restart the notebook (one time only).

python

!git clone https://github.com/davidtvs/pytorch-lr-finder.git && cd pytorch-lr-finder && python setup.py install

python

import torch
from torch import nn
from typing import List
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelWithLMHead, AdamW, get_linear_schedule_with_warmup
import logging
import os
from functools import lru_cache
from tokenizers import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing
import pytorch_lightning as pl
from torch.utils.data import DataLoader, Dataset
import pandas as pd
from argparse import Namespace
from sklearn.metrics import classification_report
torch.__version__

Understanding pre-trained language model input / outputs

In order to build your classifier on top of pre-trained language model you must first understand it outputs. Usually it requries reading the related paper or (if possible) reading the documentation / github code. Fortunately, it’s easy for Transformers library - as the models are documented and the return value is described well. For my case (RoBERTa) the docs are here: https://huggingface.co/transformers/model_doc/roberta.html#robertamodel.

We’re interested what RobertaForMaskedLM returns. It this case, output format is the following:

python

sequence_output, pooled_output, (hidden_states), (attentions)

Let’s take a look at it ourselves:

python

tokenizer = AutoTokenizer.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1")
model = AutoModelWithLMHead.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1")

We are actually interested only in base model, without the LM head which is:

python

base_model = model.roberta

Let’s see what does output look like by running the model forward pass with some input. We need to tokenize the input first:

python

text = "Mówiąc otwarcie jesteście reliktami. Ilu wiedźminów jeszcze żyje? Kilku? Góra kilkunastu. Nie stanowicie k<mask>. Poza tym, Zakon nie każe sobie płacić za ratowanie ludzi od złego."
enc = tokenizer.encode_plus(text)
enc.keys()

dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])

Tokenizer returns input_ids which are the tokens of your input text. Then token_type_ids and attention_mask - first one is used when you use LM for next-sentence prediction task, second one is to tell the model which tokens are from your input text and which are just padding tokens.

For classification, you only need input_ids and attention_mask as token types are created on the fly by the internal model’s forward function (source).

Let’s run the model. The .unsqueeze(0) is just to simulate batch processing (in this case, it will be batch size of 1).

python

# sequence_output, pooled_output, (hidden_states), (attentions)
out = base_model(torch.tensor(enc["input_ids"]).unsqueeze(0), torch.tensor(enc["attention_mask"]).unsqueeze(0))
out[0].shape, out[1].shape

(torch.Size([1, 12, 768]), torch.Size([1, 768]))

We are interested only in first element of the tuple while building classifier as the second element is used for next-sentence prediction task and it is literally said in the Transformers docs not to use it:

This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence. [source]

Output shape description:

Keep in mind that for bigger batch sizes you have to pad to either longest text’s lenght or to model’s max supported length (which usually for BERT-like models is 512).

The second dimension contains internal representation of each of the input tokens, so for properly tokenized data, i.e:

python

t = "Mówiąc otwarcie jesteście reliktami."
enc = tokenizer.encode_plus(t)
token_representations = base_model(torch.tensor(enc["input_ids"]).unsqueeze(0))[0][0]
print(enc["input_ids"])
print(tokenizer.decode(enc["input_ids"]))
print(f"Length: {len(enc['input_ids'])}")
print(token_representations.shape)

[0, 355, 304, 1748, 71, 12029, 28398, 807, 2578, 19022, 18, 2]
<s> Mówiąc otwarcie jesteście reliktami.</s>
Length: 12
torch.Size([12, 768])

Representation for each token is as follows:

0 = <s> token is represented by token_representations[0]
355 = M token is represented by token_representations[1]

and so on…

Build custom classification head on top of LM

OK, we understand the model’s output and sizes, now we can build custom classification head on top of it. Have you found new awesome activation function? Or maybe you want to try-out different pooling methods to better fit your case? You’re free to experiment. In this example, I will try out custom Mish activation function on top of straightforward classification head inspired by RoBERTa’s original one (source) and DistilBERT’s custom initialization. Because why not!

python

# from https://github.com/digantamisra98/Mish/blob/b5f006660ac0b4c46e2c6958ad0301d7f9c59651/Mish/Torch/mish.py
@torch.jit.script
def mish(input):
    return input * torch.tanh(F.softplus(input))
  
class Mish(nn.Module):
    def forward(self, input):
        return mish(input)

python

class PolBERTaSentimentModel(nn.Module):
    def __init__(self, base_model, n_classes, base_model_output_size=768, dropout=0.05):
        super().__init__()
        self.base_model = base_model
        
        self.classifier = nn.Sequential(
            nn.Dropout(dropout),
            nn.Linear(base_model_output_size, base_model_output_size),
            Mish(),
            nn.Dropout(dropout),
            nn.Linear(base_model_output_size, n_classes)
        )
        
        for layer in self.classifier:
            if isinstance(layer, nn.Linear):
                layer.weight.data.normal_(mean=0.0, std=0.02)
                if layer.bias is not None:
                    layer.bias.data.zero_()

    def forward(self, input_, *args):
        X, attention_mask = input_
        hidden_states, _ = self.base_model(X, attention_mask=attention_mask)
        
        # here I use only representation of <s> token, but you can easily use more tokens,
        # maybe do some pooling / RNNs... go crazy here!
        return self.classifier(hidden_states[:, 0, :])

Hint

As a sanity check, I encourage you to run forward pass of the model on dummy data to validate the code. It also better to do this on CPU, as the stack traces of potential exceptions will be more meaningful. Mysterious CUDA erros very often are caused by simple Python-level errors.

python

classifier = PolBERTaSentimentModel(AutoModelWithLMHead.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1").roberta, 3)

python

X = torch.tensor(enc["input_ids"]).unsqueeze(0)
attn = torch.tensor(enc["attention_mask"]).unsqueeze(0)

python

classifier((X, attn))

tensor([[0.5643, 0.2237, 0.1142]], grad_fn=<AddmmBackward>)

The output is correct - we’ve run the model with batch size and expected 3-class output (for CrossEntropyLoss).

Preparing fast tokenizers to use with text dataset

When you prepare your dataset you have two options while implementing Dataset class:

return tokenized text and class in __getitem__ function
return just text and class in __getitem__ function and implement Collate function - this function will be used by PyTorch DataLoader to prepare batches

Here I’ve chosen the second approach.

Default tokenizer loaded above (as for Transformers v2.5.1) uses Python implementation. In order to leverage full potential of parallel Rust tokenizers, we need to save the tokenizer’s internal data and then create instance of fast tokenizer with it.

python

!mkdir -p tokenizer

python

tokenizer.save_pretrained("tokenizer")

('tokenizer/vocab.json',
 'tokenizer/merges.txt',
 'tokenizer/special_tokens_map.json',
 'tokenizer/added_tokens.json')

python

!ls tokenizer

merges.txt  special_tokens_map.json  tokenizer_config.json  vocab.json

Once saved, implement CollateFn using fast tokenizers. As you can see, the whole batch can be tokenized in parallel and the tokenizer will also automatically return attention mask and pad input to max length of 512.

Important!

Ensure that all of the special tokens are in appropriate places and their IDs come from the actual tokenizer you’re using - they should not be hardcoded, as different models will have different IDs for padding, BOS, EOS and so on.

python

class TokenizersCollateFn:
    def __init__(self, max_tokens=512):
        t = ByteLevelBPETokenizer(
            "tokenizer/vocab.json",
            "tokenizer/merges.txt"
        )
        t._tokenizer.post_processor = BertProcessing(
            ("</s>", t.token_to_id("</s>")),
            ("<s>", t.token_to_id("<s>")),
        )
        t.enable_truncation(max_tokens)
        t.enable_padding(max_length=max_tokens, pad_id=t.token_to_id("<pad>"))
        self.tokenizer = t

    def __call__(self, batch):
        encoded = self.tokenizer.encode_batch([x[0] for x in batch])
        sequences_padded = torch.tensor([enc.ids for enc in encoded])
        attention_masks_padded = torch.tensor([enc.attention_mask for enc in encoded])
        labels = torch.tensor([x[1] for x in batch])
        
        return (sequences_padded, attention_masks_padded), labels

Getting data for classification

Here I use PolEmo 2.0 dataset, which is a benchmark for sentiment analysis task in Polish language.

python

!wget -O dataset_conll.zip https://clarin-pl.eu/dspace/bitstream/handle/11321/710/dataset_conll.zip?sequence=1&isAllowed=y

python

!unzip -n dataset_conll.zip

Archive:  dataset_conll.zip

python

train_path, val_path, test_path = [f"dataset_conll/all.text.{d}.txt" for d in ("train", "dev", "test")]
train_path, val_path, test_path

('dataset_conll/all.text.train.txt',
 'dataset_conll/all.text.dev.txt',
 'dataset_conll/all.text.test.txt')

This dataset has CSV-like format with __label__ text being the separator. Don’t ask. Let’s take a quick look at the data:

python

df = pd.read_csv(train_path, sep="__label__", header=None, names=["text", "label"], engine="python")
df.head()

	text	label
0	Na samym wejściu hotel śmierdzi . W pokojach j...	meta_minus_m
1	Ze swoimi dziećmi ( 10 lat i 2 latka ) chodzę ...	meta_plus_m
2	Chciał em zestaw do podstawowych prac coś przy...	meta_minus_m
3	Drogie Mamy . Mam o tym laktatorze takie same ...	meta_minus_m
4	własciwie nic się nie dowidział em , podczas b...	meta_minus_m

python

df.label.unique()

array(['meta_minus_m', 'meta_plus_m', 'meta_zero', 'meta_amb'],
      dtype=object)

You will need mapping between text label and integer:

python

label2int = {
  "meta_minus_m": 0,
  "meta_plus_m": 1,
  "meta_zero": 2,
  "meta_amb": 3
}

As you can see below, the Dataset itself is really simple as it returns only the text and label. The rest is handled by the TokenizersCollateFn above.

python

class PolEmo2Dataset(Dataset):
    def __init__(self, path):
        super().__init__()
        self.data_column = "text"
        self.class_column = "class"
        self.data = pd.read_csv(path, sep="__label__", header=None, names=[self.data_column, self.class_column],
                               engine="python")

    def __getitem__(self, idx):
        return self.data.loc[idx, self.data_column], label2int[self.data.loc[idx, self.class_column]]

    def __len__(self):
        return self.data.shape[0]

Sanity check:

python

ds = PolEmo2Dataset(train_path)
ds[13]

('Była m przekonana , że w Sopocie jest jeden hotel , który może zapewnić najlepszą jakość , tym większe było moje zaskoczenie krótkim pobytem służbowym w Małej Anglii . Miała m okazję mieszkać w pięknym i wygodnym pokoju o bardzo wysokim standardzie . Przemiła obsługa , co jest kluczowe i często niesieciowe hotele nie potrafią sobie z tym poradzić , tutaj nie było takiego problemu . Bardzo przyjemna strefa wellness oraz spa . Doskonała restauracja śniadaniowa , przepięknie urządzona , śniadanie bez zarzutu i kawa . . doskonała , każdy kawosz zrozumie . Brak widoku na morze został zupełnie zrekompensowany . Polecam i mam nadzieję , że taki poziom już pozostanie . Dziękuję za miłe wspomnienia . ',
 1)

Preparing reproducible training code with PyTorch Lightning

Most recently, PyTorch Lightning is my go to library for quick prototyping with PyTorch. You have clear API that is actually extension of the original PyTorch nn.Module one with all of the repeatable parts like training loop, validation loop, using GPUs, learning rate schedulers, gradient accumulation, tensorboard, checkpointing and many others handled for you.

There is only a thin API you have to implement, which is pl.LightningModule. Required parts are:

__init__ to initialize custom variables in your model
forward same as you would do in nn.Module - here I wrap my model inside of the pl.LightningModule but you could as well implement it directly here
training_step - take care of the forward pass for the model and return loss
validation_step and test_step - same as above but for the different learning phases
train_dataloader - DataLoader for your training dataset
val_dataloader and test_dataloader - same as above but for the different learning phases
configure_optimizers - prepare optimizer and (optionally) LR scheudles

python

class TrainingModule(pl.LightningModule):
    def __init__(self, hparams):
        super().__init__()
        self.model = PolBERTaSentimentModel(AutoModelWithLMHead.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1").roberta, 4)
        self.loss = nn.CrossEntropyLoss()
        self.hparams = hparams

    def step(self, batch, step_name="train"):
        X, y = batch
        loss = self.loss(self.forward(X), y)
        loss_key = f"{step_name}_loss"
        tensorboard_logs = {loss_key: loss}

        return { ("loss" if step_name == "train" else loss_key): loss, 'log': tensorboard_logs,
               "progress_bar": {loss_key: loss}}

    def forward(self, X, *args):
        return self.model(X, *args)

    def training_step(self, batch, batch_idx):
        return self.step(batch, "train")
    
    def validation_step(self, batch, batch_idx):
        return self.step(batch, "val")

    def validation_end(self, outputs: List[dict]):
        loss = torch.stack([x["val_loss"] for x in outputs]).mean()
        return {"val_loss": loss}
        
    def test_step(self, batch, batch_idx):
        return self.step(batch, "test")
    
    def train_dataloader(self):
        return self.create_data_loader(self.hparams.train_path, shuffle=True)

    def val_dataloader(self):
        return self.create_data_loader(self.hparams.val_path)

    def test_dataloader(self):
        return self.create_data_loader(self.hparams.test_path)
                
    def create_data_loader(self, ds_path: str, shuffle=False):
        return DataLoader(
                    PolEmo2Dataset(ds_path),
                    batch_size=self.hparams.batch_size,
                    shuffle=shuffle,
                    collate_fn=TokenizersCollateFn()
        )
        
    @lru_cache()
    def total_steps(self):
        return len(self.train_dataloader()) // self.hparams.accumulate_grad_batches * self.hparams.epochs

    def configure_optimizers(self):
        optimizer = AdamW(self.model.parameters(), lr=self.hparams.lr)
        lr_scheduler = get_linear_schedule_with_warmup(
                    optimizer,
                    num_warmup_steps=self.hparams.warmup_steps,
                    num_training_steps=self.total_steps(),
        )
        return [optimizer], [{"scheduler": lr_scheduler, "interval": "step"}]

Hints and comments

hparams from __init__ is a Namespace-type object, which holds all of the models hyperparameters; PyTorch Lightning takes good care of them when checkpointing the model
*_step functions can be implemented with the same method with argument
*_dataloader functions can also be implemented in this way
in configure_optimizers the line: [{"scheduler": lr_scheduler, "interval": "step"}] takes care of the step-based LR scheduler - it does LR warmup and then linear decay, implementation comes from Transformers library

The code above is fairly simple and we don’t need anything more here. For more robust examples I send you to PyTorch Lightning docs.

Finding good starting learning rate for the model

There is a method of approximating “good” learning rate for the training. I first encountered it in Fast.ai Deep Learning course. This approach is called LR-Finder and boils down to plotting loss of the training batch (starting from really small LR) while increasing learning rate up to the poit where the loss blows up. Then, the “good” LR is one of the values that appears when the Loss/Learning Rate curve is decreasing.

Important!

Keep in mind that this method does not guarantee to provide good learning rate for every dataset. It’s just a method discovered to often work in practice.

Here I use implementation for PyTorch from here (installed at the top of this notebook).

You need to specify initial learning rate and number of steps. Other parameters are optional.

python

from torch_lr_finder import LRFinder
hparams_tmp = Namespace(
    train_path=train_path,
    val_path=val_path,
    test_path=test_path,
    batch_size=8,
    warmup_steps=100,
    epochs=3,
    lr=lr,
    accumulate_grad_batches=1,
)
module = TrainingModule(hparams_tmp)
criterion = nn.CrossEntropyLoss()
optimizer = AdamW(module.parameters(), lr=5e-7)
lr_finder = LRFinder(module, optimizer, criterion, device="cuda")
lr_finder.range_test(module.train_dataloader(), end_lr=100, num_iter=100, accumulation_steps=hparams_tmp.accumulate_grad_batches)
lr_finder.plot()
lr_finder.reset()

Stopping early, the loss has diverged
Learning rate search finished. See the graph with {finder_name}.plot()

From the plot above we can guess that something between 1e-5 and 1e-4 would be a good learning rate, as everyhing higher results in increased loss.

To ensure the right spot was chosen, you can plot it:

python

lr = 4e-5
lr

4e-05

python

lr_finder.plot(show_lr=lr)

Training the sentiment analysis classifier on PolEmo2.0

We know the learning rate, we have the code for the model. Now it’s time to run it.

python

hparams = Namespace(
    train_path=train_path,
    val_path=val_path,
    test_path=test_path,
    batch_size=8,
    warmup_steps=100,
    epochs=3,
    lr=lr,
    accumulate_grad_batches=1
)
module = TrainingModule(hparams)

python

# just to get rid of any leftovers before the training - useful when you run different cells multiple times
import gc; gc.collect()
torch.cuda.empty_cache()

Using Trainer from PyTorch Lightning

The pl.Trainer provides all-in-one solution for training, validation and testing. There is a lot of useful parameters for almost every aspect of training you can think of, everything is well documented here.

python

trainer = pl.Trainer(gpus=1, max_epochs=hparams.epochs, progress_bar_refresh_rate=10,
                     accumulate_grad_batches=hparams.accumulate_grad_batches)

python

trainer.fit(module)

Evaluating the trained model

As this dataset is small, I’ve used scikit-learn to get the overall classification report from the model.

Important!

Be aware that for big datasets this approach might not be optimal as it transfers the data from GPU to CPU for the whole dataset and puts every prediction in memory. For big datasets it’s better to implement metrics inside of the pl.LightningModule when the dataset is processed in batches.

Ensure that you disable gradients and put model into evaluation state (module.eval()) to disable dropouts during prediction.

python

with torch.no_grad():
    progress = ["/", "-", "\\", "|", "/", "-", "\\", "|"]
    module.eval()
    true_y, pred_y = [], []
    for i, batch_ in enumerate(module.test_dataloader()):
        (X, attn), y = batch_
        batch = (X.cuda(), attn.cuda())
        print(progress[i % len(progress)], end="\r")
        y_pred = torch.argmax(module(batch), dim=1)
        true_y.extend(y.cpu())
        pred_y.extend(y_pred.cpu())
print("\n" + "_" * 80)
print(classification_report(true_y, pred_y, target_names=label2int.keys(), digits=6))

python

    ________________________________________________________________________________
                  precision    recall  f1-score   support
    
    meta_minus_m   0.880878  0.828909  0.854103       339
     meta_plus_m   0.867521  0.894273  0.880694       227
       meta_zero   0.957983  0.966102  0.962025       118
        meta_amb   0.574324  0.625000  0.598592       136
    
        accuracy                       0.832927       820
       macro avg   0.820177  0.828571  0.823854       820
    weighted avg   0.837433  0.832927  0.834617       820

Additional links and resources

this notebook in Google Colab - https://colab.research.google.com/drive/1sajgpLTrTJDzRSlxycy8aE6ysxGqaT18
this notebook on GitHub - https://github.com/marrrcin/transformers-sentiment-analysis
PyTorch Lightning documentation - https://pytorch-lightning.readthedocs.io/en/0.7.1/

Training RoBERTa from scratch - the missing guide

Marcin Zabłocki — Sun, 15 Mar 2020 23:00:00 +0000

After hours of research and attempts to understand all of the necessary parts required for one to train custom BERT-like model from scratch using HuggingFace’s Transformers library I came to conclusion that existing blog posts and notebooks are always really vague and do not cover important parts or just skip them like they weren’t there - I will give a few examples, just follow the post.

I’ve decided to get my hands dirty and try to train transformer language model (BERT or other) for Polish language. Here you will find all of the missing pieces that I have encountered while doing it.

Many thanks to Egnyte Inc. for providing me with necessary resources to run the training on powerful machine with 4x NVIDIA P100 GPUs.

TL;DR

This post covers:

what mistakes to avoid
how to prepare own dataset with details - I use Polish Wikipedia and Wolne Lektury datasets
how to prepare training configuration
how to train new tokenizers using HugginFace’s Rust tokenizers
how to use run_language_modeling.py script from HuggingFace library to train your model - what parameters to set, how to deal with the script
how to monitor training process
how to share the model on HuggingFace’s model hub

Getting the dataset

The most important part when dealing with language models is to have solid dataset with text in the language you will be modeling. In my proof of concept I’ve started with Polish Wikipedia and then extended my dataset with Wolne Lektury (repository of public domain polish books and poems).

Preparing Polish Wikipedia dump

So there is a “just get wikipedia” part that noone covers in details but I find it important. First of all, on the wikipedia dumps page there are many files that you can download. I’ve decided to download dump from 2020-02-20 labeled as:

Articles, templates, media/file descriptions, and primary meta-pages, in multiple bz2 streams, 100 pages per stream

which consisted of 7 files with names like: plwiki-20200220-pages-articles-multistream*.xml*.bz2. Why? That kind of split allows to simpify preprocessing - I don’t have to have lots of RAM to processs smaller files. Another benefit is that I can just hold-out one or two of those files for validation phase after my language model learning is complete.

Every file is a huge XML containing articles with mediawiki-specific markup language. I used this tool to extract well formated JSONs from them: https://github.com/attardi/wikiextractor (clone the repo).

So now I have a list of files for both training and evaluation:

bash

    # train.txt
    plwiki-20200220-pages-articles-multistream1.xml-p1p169750
    plwiki-20200220-pages-articles-multistream2.xml-p169751p510662
    plwiki-20200220-pages-articles-multistream3.xml-p510663p1056310
    plwiki-20200220-pages-articles-multistream4.xml-p1056311p1831508
    plwiki-20200220-pages-articles-multistream5.xml-p1831509p3070393
    plwiki-20200220-pages-articles-multistream6.xml-p4570394p4720470

and

bash

    # eval.txt
    plwiki-20200220-pages-articles-multistream6.xml-p3070394p4570393

To extract them just run:

bash

cat train.txt | xargs -I@  python wikiextractor/WikiExtractor.py @ --bytes=100M --json --output="./plwiki-json/train/@"

After extracting you will get the following structure:

bash

├── eval
│   └── plwiki-20200220-pages-articles-multistream6.xml-p3070394p4570393
│       └── AA
│           ├── wiki_00
│           ├── wiki_01
│           ├── wiki_02
│           └── wiki_03
└── train
    ├── plwiki-20200220-pages-articles-multistream1.xml-p1p169750
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       └── wiki_02
    ├── plwiki-20200220-pages-articles-multistream2.xml-p169751p510662
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       └── wiki_02
    ├── plwiki-20200220-pages-articles-multistream3.xml-p510663p1056310
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       └── wiki_02
    ├── plwiki-20200220-pages-articles-multistream4.xml-p1056311p1831508
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       ├── wiki_02
    │       └── wiki_03
    ├── plwiki-20200220-pages-articles-multistream5.xml-p1831509p3070393
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       ├── wiki_02
    │       └── wiki_03
    └── plwiki-20200220-pages-articles-multistream6.xml-p4570394p4720470
        └── AA
            └── wiki_00

Extracting text from wikipedia

After you get JSON lines from wikipedia dump, you need to transform articles in JSON text into plaintext consumable by the language model training script.

Important!

I’ve probably made a mistake here which is important later on! I first tokenized the text by sentences and outputted each sentence in a separate line because of the ambiguity of available resources about dataset preparation. I think that it affects the quality of both tokenization and training. Do not make that mistake.

After digging throughout the HuggingFace’s github I’ve finally got an answer that there should be one document per line. OK - but what if the text is really long? We will be dealing not only with long wiki articles but later on - with books that have thousands of characters and current SOTA language models have limit of around 512-768 tokens (depends on the model). The possible solution for that is to use sliding window over sentences in the input text. For more details see the following thread: https://github.com/huggingface/transformers/issues/2693

To extract the text and tokenize the input I’ve used nltk with custom polish abbreviations - thanks to Krzysztof Sopyła’s awesome gist: https://gist.github.com/ksopyla/f05fe2f48bbc9de895368b8a7863b5c3.

You can find my extraction notebook here: https://gist.github.com/marrrcin/e383b75a5d0dad42048847d97965e037. I decided to skip it here to not pollute the post.

Key takeaways from preprocessing:

tokenize article into sentences
use sliding window over sentences to handle long texts - I used sliding window of size 4

Once the extraction completes, you will have training file like this:

bash

Sumatra Północna Sumatra Północna (indonez. "Sumatera Utara") – prowincja w Indonezji w północnej części Sumatry. Obejmuje również wyspę Nias i wyspy Batu leżące na Oceanie Indyjskim. Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan.
"Sumatera Utara") – prowincja w Indonezji w północnej części Sumatry. Obejmuje również wyspę Nias i wyspy Batu leżące na Oceanie Indyjskim. Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan. Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba.
Obejmuje również wyspę Nias i wyspy Batu leżące na Oceanie Indyjskim. Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan. Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba. Bliżej wybrzeży nizinne tereny w dużym stopniu pokryte bagnami.
Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan. Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba. Bliżej wybrzeży nizinne tereny w dużym stopniu pokryte bagnami. Ludność prowincji tworzą Malajowie wyznający islam, Batakowie będący w większości chrześcijanami (gł. protestantami) oraz przybysze z Jawy, Chin, Indii.
Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba. Bliżej wybrzeży nizinne tereny w dużym stopniu pokryte bagnami. Ludność prowincji tworzą Malajowie wyznający islam, Batakowie będący w większości chrześcijanami (gł. protestantami) oraz przybysze z Jawy, Chin, Indii. Główne miasta: Medan, Binjai, Pematang Siantar, Tebing Tinggi

Prepare Wolne Lektury dump

For extraction of polish books I have crawled https://wolnelektury.pl/katalog/ site. It’s pretty straightforward process - all of the books can be downloaded in well formatted TXT files. The preprocessing steps are exactly the same as for wiki, namely: tokenize book into sentences, use sliding window to transform long books into smaller chunks without dropping any data - this time I used sliding window of 8 sentences. Gist of preprocessing notebook: https://gist.github.com/marrrcin/bcc115fbadf79eba9d9c8ca711da9e20.

Training new tokenizer

At this point, I’ve decided to go with RoBERTa model. Model you choose determines the tokenizer that you will have to train. For RoBERTa it’s a ByteLevelBPETokenizer, for BERT it would be BertWordPieceTokenizer (both from tokenizers library). Training the tokenizer is super fast thanks to the Rust implementation that guys at HuggingFace have prepared (great job!).

python


from pathlib import Path

from tokenizers import ByteLevelBPETokenizer
from glob import glob
paths = list(
    glob("your_input_dataset_files/*.txt")
)
# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer(lowercase=False)

# Customize training
tokenizer.train(files=paths, vocab_size=32000, min_frequency=3, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

# Save files to disk
import os
OUT_DIR = "polish_tokenizer_bpe_32k"
os.makedirs(OUT_DIR, exist_ok=True)
tokenizer.save(OUT_DIR, "pl")

Here, the special_tokens part is super important - those tokens need to be exactly in the same order, because otherwise the model will crash!.

For BERT tokenizer however, you will have:

python

from tokenizers import BertWordPieceTokenizer, ByteLevelBPETokenizer
# Initialize a tokenizer
tokenizer = BertWordPieceTokenizer(lowercase=False, handle_chinese_chars=False)

Adjusting the `run_language_modeling.py` script

The version I used and modified comes from v.2.5.1 release of HuggingFace transformers: https://github.com/huggingface/transformers/blob/v2.5.1/examples/run_language_modeling.py

Dataset - check, tokenizer - check. Now it’s time for the model training. Initially, I wanted to use the run_language_modeling.py script from HuggingFace git repo without changes, but it’s it does not support fast Rust-based tokenizers (as for 2020-03-16). Here I mention the minimal changes I made to make it work.

Fast tokenizers support in `run_language_modeling.py`

In order to plug fast tokenizer I’ve trained above into this script I had to modify the LineByLineTextDataset that’s provided there. The final version looks like this:

python

class LineByLineTextDataset(Dataset):
    def __init__(self, t: PreTrainedTokenizer, args, file_path: str, block_size=512):
        assert os.path.isfile(file_path)
        logger.info("Creating features from dataset file at %s", file_path)
        
        # -------------------------- CHANGES START
        bert_tokenizer = os.path.join(args.tokenizer_name, "vocab.txt")
        if os.path.exists(bert_tokenizer):
            logger.info("Loading BERT tokenizer")
            from tokenizers import BertWordPieceTokenizer
            tokenizer = BertWordPieceTokenizer(os.path.join(args.tokenizer_name, "vocab.txt"), handle_chinese_chars=False, lowercase=False)
            tokenizer.enable_truncation(512)
        else:
            from tokenizers import ByteLevelBPETokenizer
            from tokenizers.processors import BertProcessing
            logger.info("Loading RoBERTa tokenizer")
            
            tokenizer = ByteLevelBPETokenizer(
                os.path.join(args.tokenizer_name, "vocab.json"),
                os.path.join(args.tokenizer_name, "merges.txt")
            )
            tokenizer._tokenizer.post_processor = BertProcessing(
                ("</s>", tokenizer.token_to_id("</s>")),
                ("<s>", tokenizer.token_to_id("<s>")),
            )
            tokenizer.enable_truncation(max_length=512)

        logger.info("Reading file %s", file_path)
        with open(file_path, encoding="utf-8") as f:
            lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]

        logger.info("Running tokenization")
        self.examples = tokenizer.encode_batch(lines)
        
        # -------------------------- CHANGES END

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        return torch.tensor(self.examples[i].ids, dtype=torch.long)

So the only changes are in the way of processing input args.tokenizer_name parameter - if the folder has vocab.txt it’s BERT tokenizer, RoBERTa otherwise. Thanks to the tokenizer.encode_batch(lines) the whole dataset is tokenized in parallel.

Important!

If you don’t have enough RAM to fit the whole dataset into memory, you have to implement your own dataset. My input dataset - Polish wikipedia with sliding window of 4 + Wolne Lektury with sliding window of 8 took ~235GB of RAM.

Preparing training configuration

This part is the most tricky one, as there are a lot of magic numbers in the posts / notebooks I’ve found, even in the official one https://huggingface.co/blog/how-to-train. They just skip it… Start with creating the directory for the configuration, say MyRoBERTaConfig. Copy the tokenizer files and rename them. Be sure to finally have:

bash

    config.json
    tokenizer_config.json
    merges.txt
    vocab.json

in this directory.

Training configuration for RoBERTa

You know what? You don’t have to specify any model-specific parameters if you want to go with the defaults! The only thing that you actually need to specify is the vocabulary size, because it determines the output layer size (Language Model is predicting the tokens after all…). For my RoBERTa I have the following config.json:

javascript

{"architectures": ["RobertaForMaskedLM"], "max_position_embeddings": 514, "vocab_size": 32000}

The architectures is self explanatory - this is the language model you will be training, vocab_size comes from the tokenizer you have created.

Important!

The tricky one is max_position_embeddings - no one will tell you that, but if you don’t set this here, the script will just crash. Awesome. I believe that for BERT model it’s not required.

Tokenizer configuration for RoBERTa is simple (tokenizer_config.json). However it’s probably not required, because we’ve overridden the LineByLineTextDataset dataset with tokenizer.

javascript

{"max_len": 512}

Training your own RoBERTa language model

Here I will show two types of run_language_modeling.py script execution - one for training from scratch and the second - for fine tuning.

Training from scratch

This is the usual case for any new language model. First, the script:

bash

#!/bin/bash
export TRAIN_FILE=plwiki.train.txt
export EVAL_FILE=plwiki.eval.txt
# run_language_modeling_with_tokenizers.py -- it's the version with support for fast tokenizers, see above
python run_language_modeling_with_tokenizers.py \
    --train_data_file $TRAIN_FILE \
    --eval_data_file $EVAL_FILE \
    --output_dir ./MyRoBERTa \
    --model_type roberta \
    --mlm \
    --config_name ./MyRoBERTaConfig \
    --tokenizer_name ./MyRoBERTaConfig \
    --do_train \
    --do_eval \
    --line_by_line \
    --learning_rate 1e-5 \
    --num_train_epochs 5 \
    --save_total_limit 20 \
    --save_steps 5000 \
    --per_gpu_train_batch_size 8 \
    --warmup_steps=10000 \
    --logging_steps=100 \
    --gradient_accumulation_steps=4 \
    --seed 666 --block_size=512

Arguments explanation: (only the non-trivial)

--output_dir - here you will get all of the model checkpoints, saved as often as specified in --save_steps argument
--mlm - configure the script to run Masked Language Model task. The script will fortunately yell at you if the --model_type is not appropriate for this task
--line_by_line - we say to use LineByLineTextDataset, the script will use my overridden version with tokenizers
--save_total_limit - how many checkpoint to keep on the disk - the oldest one will be removed once this limit is exceeded. It’s good to set it here in order to not clog your disk
--per_gpu_train_batch_size - setting this parameter might require a few failed runs fro you. If you don’t know how big batch size you can fit into your GPU, just sample your training data, to say 10k lines and launch the script a few times. Monitor the GPU usage by using watch -n 1 nvidia-smi command. Set it to as high number as possible. For Nvidia P100 and 512 --block_size, it’s 8.
--gradient_accumulation_steps - this parameter configures how many steps you will pass in forward pass of the network before running backward pass, it’s useful when you GPU cannot handle large batches
--warmup_steps - configures learning rate scheduler; as per BERT paper recommendation - it’s good to do warmup and then warmdown. Number of steps depends on your training dataset size
--logging_steps - tells how often to store Tensorboard logs - the script only outputs loss and learning rate

Training from checkpoint (finetuning)

After I’ve pretrained my model on wikipedia only, I decided to finetune it further on the Wolne Lektury dataset. Training script has produced a few snapshots, from which every one has the following structure:

bash

├── checkpoint-20000
│   ├── config.json
│   ├── merges.txt
│   ├── optimizer.pt
│   ├── pytorch_model.bin
│   ├── scheduler.pt
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   ├── training_args.bin
│   └── vocab.json

Important!

If you’re changing the input dataset, you cannot use existing scheduler.pt and optimizer.pt as you’re not “resuming” the training. You should remove (but backup first!) those files if you want to fine tune on another dataset.'

OK, now the script itself:

bash

#!/bin/bash
export TRAIN_FILE=wolne-lektury.sliding8.txt
export EVAL_FILE=plwiki.eval.txt

python run_language_modeling_with_tokenizers.py \
    --train_data_file $TRAIN_FILE \
    --eval_data_file $EVAL_FILE \
    --output_dir ./MyRoBERTa_with_another_dataset \
    --model_type roberta \
    --model_name_or_path ./MyRoBERTa/checkpoint-30000 \
    --mlm \
    --config_name ./MyRoBERTaConfig \
    --tokenizer_name ./MyRoBERTaConfig \
    --do_train \
    --do_eval \
    --line_by_line \
    --learning_rate 5e-5 \
    --num_train_epochs 3 \
    --save_total_limit 20 \
    --save_steps 5000 \
    --per_gpu_train_batch_size 8 \
    --warmup_steps=5000 \
    --logging_steps=100 \
    --gradient_accumulation_steps=4 \
    --mlm_probability=0.2 \
    --seed 666 --block_size=512

Arguments - notable mentions:

--mlm_probability=0.2 - this parameter controls the percentage of the tokens you mask during training; default is 0.15, I’ve decided to change it to make the training more difficult to the model
--model_name_or_path ./MyRoBERTa/checkpoint-30000 - this should point to the checkpoint of your previously trained model; remember the note above!

Training phase

I highly recommend you to launch the script using screen tool, which allows to easily spawn and manage multiple SSH sessions - you will be able to disconnect from the server while the training runs. And it will run for days… so brace yourself. You can use screen -UL command (-U for enabling UTF-8 mode for progress bars and -L to log session output to file - it’s helpful when something crashes).

During training, the script will output Tensorboard logs into runs directory. You can monitor them by running tensorboard command:

bash

tensorboard --logdir=runs

It will launch small webserver on localhost:6006 and you will be able to monitor the training.

Training summary for Polish RoBERTa a.k.a PolBERTa

I’ve run my training in three phases:

From scratch on Polish Wikipedia only (1 sentence per line) for 370k steps using learning rate 1e-5 with 10k warmup steps.
Fine tuning on Wolne Lektury only (8 sentences per line) for 60k steps, starting after checkpoint from 1. Learning rate 5e-5 with 5k warmup steps.
Fine tuning on Wolne Lektury (8 sentences per line) and Polish Wikipedia (4 sentences per line) for 30k steps. Learning rate 5-e5 with 10k warmup steps.

Tensorboard for all of the runs can be viewed here https://tensorboard.dev/experiment/aYrAE9uMTxGKRtLUOFjprg/

Exporting the model

If you want to share the model with the NLP community (which I highly encourage you to do!) you need to export it in appropriate format. Prepare the path for your model’s latest checkpoint and then run the following code:

python

from transformers import AutoModelWithLMHead, AutoTokenizer
import os
directory = "/path/to/your/model/checkpoint-30000"
model = AutoModelWithLMHead.from_pretrained(directory)
tokenizer = AutoTokenizer.from_pretrained(directory)
out = "MyBERTa-base-cased-v1"
os.makedirs(out, exist_ok=True)
model.save_pretrained(out)
tokenizer.save_pretrained(out)

Then use transformers-cli util to upload the model:

bash

transformers-cli upload ./MyBERTa-base-cased-v1/

Using my pre-trained model

I’ve uploaded my pre-trained RoBERTa to HuggingFace’s model hub: marrrcin/PolBERTa-base-polish-cased-v1 Usage:

python

from transformers import AutoModelWithLMHead, AutoTokenizer, pipeline

model = AutoModelWithLMHead.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1")
tokenizer = AutoTokenizer.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1")

fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
fill_mask("Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad <mask>, a postępujesz tak jakbyś miał niecałe dziesięć.")

Output:

javascript

[{'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad sto, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.1013106107711792,
  'token': 1675},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad tysiąc, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.09420681744813919,
  'token': 31144},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad połowę, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.0891065001487732,
  'token': 13627},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad dwadzieścia, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.07204979658126831,
  'token': 18811},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad godzinę, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.059559762477874756,
  'token': 20775}]

Follow up and TODO

Here are a few follow up things and my planned future work on Polish RoBERTa:

prepare model card for HuggingFace's model hub
evaluate PolBERTa on PolEmo2.0 and other downstream tasks
re-train PolBERTa again, using bigger vocab size for tokenizer
fine tune model further on another open datasets

If you find any bug in the post or something is not clear - feel free to post a comment, I will be glad to improve this short practical guide.

Thanks to…

I would like to thank Egnyte Inc. for providing me with required resources to train the language model and also Darek Kłeczek (creator of original PolBERT) for tips related to training BERT.

Additional links and resources

PolBERT - original BERT-base-uncased LM trained by Darek Kłeczek
HuggingFaces original guide for training new language models

Cookies

Marcin Zabłocki — Wed, 06 Feb 2019 10:08:01 +0100

To make this site work properly, we sometimes place small data files called cookies on your device. Most big websites do this too.

What are cookies?

A cookie is a small text file that a website saves on your computer or mobile device when you visit the site. It enables the website to remember your actions and preferences (such as login, language, font size and other display preferences) over a period of time, so you don’t have to keep re-entering them whenever you come back to the site or browse from one page to another.

How this website uses cookies?

This website uses cookies in order to:

gather statistics
remember custom preferences
remember if you've already agreed to use cookies

How to control cookies?

You can control and/or delete cookies as you wish – for details, see aboutcookies.org. You can delete all cookies that are already on your computer and you can set most browsers to prevent them from being placed. If you do this, however, you may have to manually adjust some preferences every time you visit a site and some services and functionalities may not work.

About Me

Marcin Zabłocki — Wed, 06 Feb 2019 10:07:57 +0100

Hi, I’m Marcin

new technology lover,
who writes code over and over,
also a guy who seeks for a challenge
and isn't it a little strange,
when you seek for a CV,
I keep rhyming about me?

To be more serious - I am Staff MLOps Engineer at Printify and also a Poznań University of Technology graduate (M.Sc.). You can find detailed information about me on the social sites linked at the bottom of the page. To get in touch with me, check the contact page or just write me a tweet.

Contact

Marcin Zabłocki — Wed, 06 Feb 2019 10:07:38 +0100

To contact me, just go to one of the social portals or write me e-mail: marcin[at]<this domain>

Using ResNet for MNIST in PyTorch

Marcin Zabłocki — Sun, 06 Jan 2019 23:00:00 +0000

During last year (2018) a lot of great stuff happened in the field of Deep Learning. One of those things was the release of PyTorch library in version 1.0. PyTorch is my personal favourite neural network/deep learning library, because it gives the programmer both high level of abstraction for quick prototyping as well as a lot of control when you want to dig deeper. Alongside that, PyTorch does not force you into learning any new API conventions, because everything that you define in PyTorch - from the network architecture, throught data loading to custom loss functions is defined in plain Python, using either ordinary functions or object oriented style.

In this post I will show you how to get started with PyTorch by explaining how to use pre-defined ResNet architecture to create image classifier for the MNIST dataset. I assume that you have some minimal prior knowledge of Neural Networks and Deep Learning.

This post is deprecated, see it's updated version:
https://zablo.net/blog/post/pytorch-resnet-mnist-jupyter-notebook-2021/

TL;DR

Quick start for PyTorch by showing how to adjust ResNet architecture to create deep learning image classifier for MNIST dataset with 99% accuracy/precision/recall after only 5 epochs.

Do not reinvent the wheel

When it comes to image classification using deep learning, there was already a lot of research done in the terms of finding the best network architecture. In the famous ImageNet competition, there were year-to-year improvements and competitors came up with new ideas every time. One of the architectures that became very succesfull in ImageNet competition was ResNet (2015). I will show you how to take existing network definition from torchvision library and tweak for use with MNIST dataset.

Digging into the ResNet

ResNet were originally designed for ImageNet competition, which was a color (3-channel) image classification task with 1000 classes. MNIST dataset howerver only contains 10 classes and it’s images are in the grayscale (1-channel). So there are two things to change in the original network.

I will take ResNet18 from torchvision library (official PyTorch module with network architectures, image transformations and others). Let’s see how it’s implemented there:

python

# from: https://github.com/pytorch/vision/blob/21153802a3086558e9385788956b0f2808b50e51/torchvision/models/resnet.py#L167

def resnet18(pretrained=False, **kwargs):
    """Constructs a ResNet-18 model.
    Args:
        pretrained (bool): If True, returns a model pre-trained on ImageNet
    """
    model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)
    if pretrained:
        model.load_state_dict(model_zoo.load_url(model_urls['resnet18']))
    return model

So they just create object of class torchvision.models.resnet.ResNet with appropriate parameters. Now, what happens when we train the network? In PyTorch, the forward function of network class is called - it represent forward pass of data through the network. ResNets forward looks like this:

python

# from: https://github.com/pytorch/vision/blob/21153802a3086558e9385788956b0f2808b50e51/torchvision/models/resnet.py#L149

def forward(self, x):
    x = self.conv1(x)
    x = self.bn1(x)
    ## ... skipped a few lines ...
    return x

So the first layer (input) is conv1. It’s defined in the ResNet constructor like this:

python

# from: https://github.com/pytorch/vision/blob/21153802a3086558e9385788956b0f2808b50e51/torchvision/models/resnet.py#L104

self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)

As you can see, ResNet takes 3-channel (RGB) image. Standard input image size for this network is 224x224px.

Adjusting ResNet for MNIST

Now that we know what’s inside the ResNet, let’s adjust it for our needs.

python

class MnistResNet(ResNet):
    def __init__(self):
        super(MnistResNet, self).__init__(BasicBlock, [2, 2, 2, 2], num_classes=10)
        self.conv1 = torch.nn.Conv2d(1, 64, 
            kernel_size=(7, 7), 
            stride=(2, 2), 
            padding=(3, 3), bias=False)

I’ve changed input layer, to take single-channel image and set the number of classes to 10.

And it’s done. We have MNIST-ready ResNet network architecture.

PyTorch training loop boilerplate

Unfortunately, there is no built-in training class in the plain PyTorch library, so here is a boilerplate code for training any network (you can copy and paste it). It’s more or less based on the training loop code from official PyTorch documentation enriched with precision/recall/F1/accuracy metrics calculation, progress bar and pretty-printing.

Imports:

python

from torchvision.models.resnet import ResNet, BasicBlock
from torchvision.datasets import MNIST
from tqdm.autonotebook import tqdm
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score
import inspect
import time
from torch import nn, optim
import torch
from torchvision.transforms import Compose, ToTensor, Normalize, Resize
from torch.utils.data import DataLoader

Training loop for any PyTorch model (with evaluation)

I’ve added as many neccessary comments as possible, to ease the reading and explain confusing bits.

python

# model:
model = YourModelHere()

# params you need to specify:
epochs = 5
train_loader, val_loader = # put your data loader here
loss_function = nn.CrossEntropyLoss() # your loss function, cross entropy works well for multi-class problems

# optimizer, I've used Adadelta, as it wokrs well without any magic numbers
optimizer = optim.Adadelta(model.parameters())

start_ts = time.time()
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

losses = []
batches = len(train_loader)
val_batches = len(val_loader)

# loop for every epoch (training + evaluation)
for epoch in range(epochs):
    total_loss = 0

    # progress bar (works in Jupyter notebook too!)
    progress = tqdm(enumerate(train_loader), desc="Loss: ", total=batches)

    # ----------------- TRAINING  -------------------- 
    # set model to training
    model.train()
    
    for i, data in progress:
        X, y = data[0].to(device), data[1].to(device)
        
        # training step for single batch
        model.zero_grad()
        outputs = model(X)
        loss = loss_function(outputs, y)
        loss.backward()
        optimizer.step()

        # getting training quality data
        current_loss = loss.item()
        total_loss += current_loss

        # updating progress bar
        progress.set_description("Loss: {:.4f}".format(total_loss/(i+1)))
        
    # releasing unceseccary memory in GPU
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
    
    # ----------------- VALIDATION  ----------------- 
    val_losses = 0
    precision, recall, f1, accuracy = [], [], [], []
    
    # set model to evaluating (testing)
    model.eval()
    with torch.no_grad():
        for i, data in enumerate(val_loader):
            X, y = data[0].to(device), data[1].to(device)

            outputs = model(X) # this get's the prediction from the network

            val_losses += loss_function(outputs, y)

            predicted_classes = torch.max(outputs, 1)[1] # get class from network's prediction
            
            # calculate P/R/F1/A metrics for batch
            for acc, metric in zip((precision, recall, f1, accuracy), 
                                   (precision_score, recall_score, f1_score, accuracy_score)):
                acc.append(
                    calculate_metric(metric, y.cpu(), predicted_classes.cpu())
                )
          
    print(f"Epoch {epoch+1}/{epochs}, training loss: {total_loss/batches}, validation loss: {val_losses/val_batches}")
    print_scores(precision, recall, f1, accuracy, val_batches)
    losses.append(total_loss/batches) # for plotting learning curve
print(f"Training time: {time.time()-start_ts}s")

+ helper functions:

python

def calculate_metric(metric_fn, true_y, pred_y):
    # multi class problems need to have averaging method
    if "average" in inspect.getfullargspec(metric_fn).args:
        return metric_fn(true_y, pred_y, average="macro")
    else:
        return metric_fn(true_y, pred_y)
    
def print_scores(p, r, f1, a, batch_size):
    # just an utility printing function
    for name, scores in zip(("precision", "recall", "F1", "accuracy"), (p, r, f1, a)):
        print(f"\t{name.rjust(14, ' ')}: {sum(scores)/batch_size:.4f}")

Loading MNIST dataset and training the ResNet

One last bit is to load the data. As ResNets in PyTorch take input of size 224x224px, I will rescale the images and also normalize the numbers. Normalization helps the network to converge (find the optimum) a lot faster.

Remember to normalize the data using parameters from training dataset only, as doing otherwise (i.e calculating mean and standard deviation on the whole dataset) will cause data leak between the datasets causing the results to be incorrect!

Dataset

python

def get_data_loaders(train_batch_size, val_batch_size):
    mnist = MNIST(download=False, train=True, root=".").train_data.float()
    
    data_transform = Compose([ Resize((224, 224)),ToTensor(), Normalize((mnist.mean()/255,), (mnist.std()/255,))])

    train_loader = DataLoader(MNIST(download=True, root=".", transform=data_transform, train=True),
                              batch_size=train_batch_size, shuffle=True)

    val_loader = DataLoader(MNIST(download=False, root=".", transform=data_transform, train=False),
                            batch_size=val_batch_size, shuffle=False)
    return train_loader, val_loader

Training & evaluation

I’ve run 5 epochs on Nvidia Tesla K80 GPU. Training took ~35 minutes. Final scores are around 0.99 for precision, recall and accuracy.

Summary

Whenever you encounter image classification task, it’s better to find existing, battle tested deep neural network architecture instead of spending the time trying to build your own. The time you save on the architecture design should be spent on polishing your dataset or finding more examples.

I hope this post will help you whenever you will be building your image classifier.

Additional links & resources

Jupyter notebook for this post: https://github.com/marrrcin/pytorch-resnet-mnist
good blog post explaining the ResNets: https://medium.com/@14prakash/understanding-and-implementing-architectures-of-resnet-and-resnext-for-state-of-the-art-image-cf51669e1624

Switch-case statement in Python

Marcin Zabłocki — Sat, 20 Oct 2018 22:00:00 +0000

Have you ever wanted to use switch-case statement in Python code? If yes, then you have probably found out that there is no such expression in native Python syntax. Fortunately, you can implement it on your own while having some nerd-level fun. This post explains how to implement switch-case statement in Python.

TL;DR

How to implement switch-case statement in Python, like this one:

python

var = 666
with switch(var) as case:
    case(1, "foo")
    case(2, "yolo")
    case(3, "bar")
    case.default("dunno")
    result = case.result
print(result)
>>> "dunno"

Prerequisites & system requirements

can-do attitude
liberal approach to Python
time for some nerd-fun

Switch-Case idea in Python

Here’s the idea: Python has with statement, which can be used to control entering and exiting to the blocks of code. This is done by implementing __enter__ and __exit__ magic functions in your object.

Another magic function that is pretty handy is __call__. This one allows to handle object “calling”, which is basically this:

python

x = MyClass()
x() # <-- __call__ method of object x of type MyClass will be executed there

OK - we have our building blocks. As you’ve probably saw in the TL;DR above, our desired syntax is the following:

python

with switch(SOME_OBJECT) as case:
    case(1, "case value for 1")
    case(2, "case value for 2")
    case.default(r"¯\_(ツ)_/¯")
    result = case.result
print(result)

Additionaly, I would like to have option for case condition and case result to be lambda expression, like this:

python

y = object()
with switch(y) as case:
    case(lambda: isinstance(y, str), "Some characters")
    case(lambda: isinstance(y, int), "Numberzzzz")
    case(lambda: isinstance(y, float), "Why u not double?")
    case.default(r"¯\_(ツ)_/¯")
    result = case.result
print(result)

Switch-Case implementation in Python

Without further ado, here’s the implementation:

python

class switch(object):
    def __init__(self, object_to_switch_on):
        self.object_to_switch_on = object_to_switch_on
        self.default_result = None
        self.result_store = None
        self.cases = {}
        self.evaluated = False

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        _ = self.result

    def __call__(self, when, then):
        if when in self.cases:
            raise ValueError("This case already exists")
        else:
            self.cases[when] = then
        return self

    def default(self, then):
        self.default_result = then
        return self

    @property
    def result(self):
        if not self.evaluated:
            self.result_store = self.__evaluate__()
        return self.result_store

    def __evaluate__(self):
        from inspect import signature
        self.evaluated = True
        result_tmp = None
        if all(callable(x) for x in self.cases.keys()):
            for case_fn, case_result in self.cases.items():
                no_of_params = len(signature(case_fn).parameters)
                if no_of_params >= 2:
                    raise ValueError("Case function must have either 0 or 1 arguments")

                if no_of_params == 1 and case_fn(self.object_to_switch_on):
                    result_tmp = case_result
                    break
                elif no_of_params == 0 and case_fn():
                    result_tmp = case_result
                    break
            else:
                result_tmp = self.default_result
        elif all(not callable(x) for x in self.cases.keys()):
            result_tmp = self.cases[self.object_to_switch_on] if self.object_to_switch_on in self.cases else self.default_result
        else:
            raise ValueError("Inconsistent switch usage, use either simple objects or functions")
        if result_tmp is not None and callable(result_tmp):
            return result_tmp()
        else:
            return result_tmp

Code explanation

Class name is lowercased in order to imitate basic language syntax while used.
Construtor (__init__) takes “switchable” object.
Magic functions: __enter__ & __exit__ are here to implement Python with scope behaviour. Exit calls result property in order to trigger the evaluation when exiting the scope (it’s here just to shorten the usage syntax with lambdas even more, see exmaples below).
Magic function __call__ is used to add cases to our switch as we call it. Cases are stored internally in a dictionary for fast direct object access (as long as the case is a hashable object).
default method adds default case and it’s optional to call it.
result property getter is triggering switch-case evaluation. If it’s called for the first time, it invokes __evaluate__ method, which does all the magic of resolving switch-case statement result. Every subsequent call returns the result from cache.
__evaluate__ method is where all of the meat is. This method checks if all of the cases are either callable objects (most likely functions or lambdas). If they are, it iterates through them and assumes that match is first match for which the function/lambda returned truthy value. If none of the cases is callable, I assume that the objects are hashable and I just lookup for their values in the internal cases dictionary. After resolving the case result, I also check, whether result is callable - if it is, I invoke it, if not I return it.

Summary

And that’s it. Don’t rate the Pythonic code ratio. Like I said, it’s only for liberal Python programmes. Hope you liked it!

Additional links & resources

full project on my github: https://github.com/marrrcin/python-switch-case-statement

Use Pandas DataFrames in scikit-learn FeatureUnion and Pipelines

Marcin Zabłocki — Fri, 27 Apr 2018 22:00:00 +0000

Pandas is a popular data wrangling library among data engineers and data scientists who use Python. When it comes to machine learning with Python, scikit-learn is the top pick for writing not only Jupyter-based experiments but also for full machine learning pipelines. Unfortunately, Pandas and scikit-learn does not play well together when you try to use scikit-learn’s Pipeline and FeatureUnion abstractions. This post is here to help you deal with those problems using just a few lines of code.

TL;DR

How to use pandas DataFrame with scikit-learn FeatureUnion and Pipelines.

Environment

Python 3.6.4 (should work on 2.7.* too!)
scikit-learn 0.19.1
pandas 0.22.0

The problem

Problem with using DataFrames with scikit-learn starts to emerge when you want to preserve abilities that pandas provide i.e column names, ease of indexing, mapping and filtering. By default, scikti-learn does suport using DataFrames, however it strips them down to plain numpy arrays, which lack of programmers favourite DataFrame features.

Consider the following code:

python

raw_data = load_iris()
data = pd.DataFrame(raw_data["data"], columns=raw_data["feature_names"])

pipeline = FeatureUnion([
    ("1", make_pipeline(
        FunctionTransformer(lambda X: X.loc[:, ["sepal length (cm)"]]),
        # other transformations
    )),
    ("2", make_pipeline(
        FunctionTransformer(lambda X: X.loc[:, ["sepal width (cm)"]]),
        # other transformations
    ))
])

X = pipeline.fit_transform(data)
print(X["sepal length (cm)"].mean())
print(X["sepal width (cm)"].mean())

The idea is to apply different transformations to different columns and then merge them together for further processing / machine learning process. Looks readable and seems like it should work but if you try to run it you will get the following error:

python

FunctionTransformer(...),
AttributeError: 'numpy.ndarray' object has no attribute 'loc'

The reason for that is FunctionTransformer strips any non-numpy array objects to numpy arrays and we lose DataFrame objects there.

Solution part 1

Solution for transforming columns is simple, we need to have custom Transformer class which preserves object types:

python

class PandasTransform(TransformerMixin, BaseEstimator):
    def __init__(self, fn):
        self.fn = fn

    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None, copy=None):
        return self.fn(X)

You need to change previous FunctionTransformer to PandasTransformer:

python

pipeline = FeatureUnion([
    ("1", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal length (cm)"]]),
        # other transformations
    )),
    ("2", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal width (cm)"]]),
        # other transformations
    ))
])

X = pipeline.fit_transform(data)
print(X["sepal length (cm)"].mean())
print(X["sepal width (cm)"].mean())

It’s better this time, because the code executes the transform and variable X contains transformed data. The problem is that different exception occured:

python

IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

Why? Because scikit-learn FeatureUnion did the same stripping to DataFrames like the FunctionTransform did previously.

Solution part 2: Pandas DataFrame Featue Union

Scikit-learn team is aware of this missing feature, however GitHub issue is still unresolved. Solution for FeatureUnion problem is just to add the support of Pandas DataFrames to it.

python

class PandasFeatureUnion(FeatureUnion):
    def fit_transform(self, X, y=None, **fit_params):
        self._validate_transformers()
        result = Parallel(n_jobs=self.n_jobs)(
            delayed(_fit_transform_one)(trans, weight, X, y,
                                        **fit_params)
            for name, trans, weight in self._iter())

        if not result:
            # All transformers are None
            return np.zeros((X.shape[0], 0))
        Xs, transformers = zip(*result)
        self._update_transformer_list(transformers)
        if any(sparse.issparse(f) for f in Xs):
            Xs = sparse.hstack(Xs).tocsr()
        else:
            Xs = self.merge_dataframes_by_column(Xs)
        return Xs

    def merge_dataframes_by_column(self, Xs):
        return pd.concat(Xs, axis="columns", copy=False)

    def transform(self, X):
        Xs = Parallel(n_jobs=self.n_jobs)(
            delayed(_transform_one)(trans, weight, X)
            for name, trans, weight in self._iter())
        if not Xs:
            # All transformers are None
            return np.zeros((X.shape[0], 0))
        if any(sparse.issparse(f) for f in Xs):
            Xs = sparse.hstack(Xs).tocsr()
        else:
            Xs = self.merge_dataframes_by_column(Xs)
        return Xs

The above code seems to be long, but there are only a few things goin on there:

PandasFeatureUnion class extends scikit-learn built-in FeatureUnion
It overrides transform and fit_transform functions

Both transform and fit_transform functions were copied from scikit-learn source and the only changes were

from:

python

if any(sparse.issparse(f) for f in Xs):
    Xs = sparse.hstack(Xs).tocsr()
else:
    Xs = np.hstack(Xs)

to:

python

if any(sparse.issparse(f) for f in Xs):
    Xs = sparse.hstack(Xs).tocsr()
else:
    Xs = self.merge_dataframes_by_column(Xs)

Results

After combining PandasTransform and PandasFeatureUnion, pipeline should work like charm and output data will be still in DataFrame format.

python

pipeline = PandasFeatureUnion([
    ("1", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal length (cm)"]]),
        # other transformations
    )),
    ("2", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal width (cm)"]]),
        # other transformations
    ))
])

X = pipeline.fit_transform(data)
print(X["sepal length (cm)"].mean())
print(X["sepal width (cm)"].mean())

Summary

With just a few tricks and hacks you can use DataFrames alongside scikit-learn Pipelines to build your machine learning pipelines. I hope this post will help you!

Additional links & resources

full project on my GitHub: https://github.com/marrrcin/pandas-feature-union
scikit-learn FeatureUnion tutorial: http://scikit-learn.org/stable/auto_examples/hetero_feature_union.html

Stroke Width Transform algorithm for Python

Marcin Zabłocki — Fri, 11 Aug 2017 22:00:00 +0000

Stroke Width Transform (SWT) is a computer vision algorithm (actually, it’s an image operator) which can be used in the task of detecting text in images. This is a non-trivial task, especially for camera pictures, but SWT performs pretty well in this field.

This post is deprecated and the solution works only with Python 2.7 and old libccv versions.
For updated version (with Docker support) see: http://zablo.net/blog/post/stroke-width-transform-swt-python-3.9

TL;DR

How to wrap libccv computer vision library for using SWT in Python. Wrapper was built using SWIG.

Intro

My usecase for the SWT was the following: split big dataset of unknown images into ones that contain text (most likely documents / scans) and others. Easy, right? Yes, as long as you have fast implementation of this algorithm. Unfortunately, I needed to use Python, because the image filtering was only a part of bigger solution. I search here and there and of course, there are some implementations in Python, but they are not ready for production. One of them i.e took 5 minutes to process 2000px x 3000px image and it allocated gigabytes of memory. Just a single image. NOPE for that.

So… I searched further, but finnaly I haven’t found anything either working or performat. I decided to find any other language implementation of SWT and I’ve found libccv. Quick look at the repository - it’s written in C. OK, not my kind of language, but I gave it a try and surely, it will be fast. I’ve compiled example aplication which took image as an input and producted output in the form of 4-element tuples (x, y, width, height), each of which was an rectangle coordinates in which the text on the image is. It produced satysfying results, so I decided to use it from Python. How? Let’s dig into it!

Prerequisites

Python 2.7.* on Linux (tested on Ubuntu 16.04 x64)
Python packages: scikit-image, numpy, matplotlib (only for preview)
gcc compiler
SWIG
basic knowledge of C

SWT example breakdown from libccv

For starters, take a look at the code from https://github.com/liuliu/ccv/blob/07fc691c5344940751011c3af96d0ab202b1b4e6/bin/swtdetect.c The code comes from libccv repository and it’s an example of using SWT text detection. It is a console application, which takes file name as an argument, runs the algorithm and outputs coordinates of rectangles with text to standard output.

The probles with it are the following:

it is a console application
it takes file name as an input
it outputs to stdout

My requirements for the function (which I wanted to call from Python) were the following:

it is a standalone library / module
it takes blob of an image as an input (do not write anything to disk)
it outputs returning array-like object

SWT function wrapper in C

My requirements are pretty simple to transform to function header:

csharp

int* swt(char *bytes, int array_length, int width, int height);

Compiling the code above to plain english: take an array of bytes, it’s length (welcome to C world!), width and height of an image, do some magic and output array of numbers.

In order to implement and compile the wrapper without issues, just checkout libccv from it’s repository and put your files in libccv/lib folder. This approach is suggested by libccv author.

Implementation of SWT for Python wrapper:

csharp

#include "ccv.h"
#include <jpeglib.h>
#include "io/_ccv_io_libjpeg.c"
#include <sys/time.h>
#include <ctype.h>
#define SUCCESS 1
#define FAILURE 0

int* swt(char *bytes, int array_length, int width, int height){
    int *result_array;
    int status = FAILURE;
    ccv_enable_default_cache();
	ccv_dense_matrix_t* image = 0;

	FILE *stream;
	stream = fmemopen(bytes, array_length, "r");
	if(stream != NULL){
        int type = CCV_IO_JPEG_FILE | CCV_IO_GRAY;
        int ctype = (type & 0xF00) ? CCV_8U | ((type & 0xF00) >> 8) : 0;
        _ccv_read_jpeg_fd(stream, &image, ctype);

        if (image != 0)
        {
            ccv_array_t* words = ccv_swt_detect_words(image, ccv_swt_default_params);
            if (words)
            {
                int i;
                int result_idx = 1;
                result_array = (int*)malloc((4 * words->rnum + 1) * sizeof(int));
                result_array[0] = 4 * words->rnum;
                for (i = 0; i < words->rnum; i++)
                {
                    ccv_rect_t* rect = (ccv_rect_t*)ccv_array_get(words, i);
                    result_array[result_idx++] = rect->x;
                    result_array[result_idx++] = rect->y;
                    result_array[result_idx++] = rect->width;
                    result_array[result_idx++] = rect->height;
                }
                ccv_array_free(words);
                status = SUCCESS;
            }
            ccv_matrix_free(image);
        }
        ccv_drain_cache();
	}


    if(status != SUCCESS){
        result_array = (int*)malloc(1 * sizeof(int));
        result_array[0] = 0;
    }
    return result_array;
}

Explanation

There are some hacky things in there, I will explain them one by one.

csharp

FILE *stream;
stream = fmemopen(bytes, array_length, "r");

My function takes array of bytes as an input, but none of the ccv_read function overloads accepts this kind of data. I’ve found great fmemopen standard C function, which transforms char* buffer into file handler. But… none of the ccv_read function overloads accept this kind of data (AGAIN!). Yeah, but they read the JPEGs from files somehow!

That lead me to next part:

csharp

#include <jpeglib.h>
#include "io/_ccv_io_libjpeg.c"

In the _ccv_io_libjpeg.c file, there’s a function which accepts FILE* handler and reads JPEG image into libccv image format. I wanted to use it.

csharp

int type = CCV_IO_JPEG_FILE | CCV_IO_GRAY;
int ctype = (type & 0xF00) ? CCV_8U | ((type & 0xF00) >> 8) : 0;
_ccv_read_jpeg_fd(stream, &image, ctype);

Libccv needs to have grayscale image as an input for SWT, thus some bitwise magic to set proper flags.

csharp

int *results_array;
result_array = (int*)malloc((4 * words->rnum + 1) * sizeof(int));
result_array[0] = 4 * words->rnum;

Dynamic arrays were (and still are) such pain in C. SWT outputs 4 numbers for every match, so the size of the array should be 4 times bigger. The +1 is my hack to output everyhing in a single array and read it in Python. I used this one additional index to output the length of the whole array (I will drop this value in the upcoming SWIG wrapper, wait for it!).

csharp

if(status != SUCCESS){
    result_array = (int*)malloc(1 * sizeof(int));
    result_array[0] = 0;
}

When something goes wrong, I wanted to output something anyway, so I just create 1 element array, with 0, meaning, there is nothing to push to Python from the wrapper.

Python wrapper for SWT using SWIG

SWIG (http://www.swig.org/) seems to be really old, but it does the job pretty well (with really small amount of code). Plus it’s still active in 2017. It allows to wrap any C code into Python module really fast - that’s exactly what I wanted. You can install SWIG by using the following command:

powershell

sudo apt-get install swig

In order to build the wrapper, you need to create SWIG interface file (I named it ccvwrapper.i):

csharp

%module ccvwrapper

%{
#include "ccvwrapper.h"
%}

%typemap(out) int* swt {
	int i;
	$result = PyList_New($1[0]);
	for(i = 0; i < $1[0]; i++){
		PyObject *o = PyInt_FromLong($1[i+1]);
		PyList_SetItem($result, i, o);
	}
	free($1);
}

%include "ccvwrapper.h"

The most important path in here is the typemap delaration. This mapping converts my dynamic C array into Python list. When you want to return array from C code and use it in Python, you need to write this kind of code (this does not apply for basic types - they are automatically mapped to Python types). Header file ccvwrapper.h contains only my swt function declaration.

Building the wrapper

Just a simple bash script, following the convention from SWIG and libccv documentation. NOTE: remember to prefix output C library name with underscore _ ! SWIG requires it for building Python modules.

powershell

#!/bin/bash
swig -python ccvwrapper.i
gcc -fpic -c ccvwrapper.c ccvwrapper_wrap.c ccv_algebra.c ccv_basic.c ccv_cache.c ccv_classic.c ccv_io.c ccv_memory.c ccv_output.c ccv_resample.c ccv_sift.c ccv_swt.c ccv_transform.c ccv_util.c ./3rdparty/sha1/sha1.c -I/usr/include/python2.7

ld -shared ccvwrapper.o ccvwrapper_wrap.o ccv_algebra.o ccv_basic.o ccv_cache.o ccv_classic.o ccv_io.o ccv_memory.o ccv_output.o ccv_resample.o ccv_sift.o ccv_swt.o ccv_transform.o ccv_util.o sha1.o -ljpeg -o _ccvwrapper.so

cp ccvwrapper* ~/PycharmProjects/path_to_your_project
cp _ccvwrapper* ~/PycharmProjects/path_to_your_project

Build script uses swig command to compile interface file into two wrapepr files:

ccvwrapper_wrap.c
ccvwrapper.py

C file needs to be compiled along with the custom code and it’s dependencies. After compilation, we need to link everything into shared library with name _ccvwrapper.so. For the convenience I added two cp commands just to copy my wrapper into my project directory.

Using SWT Python wrapper

The wrapper is ready to be used. After putting all of the outputs from the build process into the project folder, you will be able to use the wrapper without any problems. Example code:

python

from __future__ import print_function
import ccvwrapper
import numpy as np
from skimage import draw
from skimage.io import imread, imshow, imsave
from matplotlib import pyplot as plt


def rectangle_perimeter(r0, c0, width, height, shape=None, clip=False):
    rr, cc = [r0, r0 + width, r0 + width, r0], [c0, c0, c0 + height, c0 + height]

    return draw.polygon_perimeter(rr, cc, shape=shape, clip=clip)


if __name__ == "__main__":
    image_name = "test_input.jpg"
    bytes = open(image_name, "rb").read()
    swt_result_raw = ccvwrapper.swt(bytes, len(bytes), 1024, 1360)
    swt_result = np.reshape(swt_result_raw, (len(swt_result_raw) / 4, 4))

    image = imread(image_name, as_grey=False)
    for x, y, width, height in swt_result:
        for i in xrange(0, 3): # just to make lines thicker
            rr, cc = rectangle_perimeter(y + i, x + i, height, width, shape=image.shape, clip=True)
            image[rr, cc] = (255, 0, 0)

    imshow(image)
    imsave("result.jpg", image)
    plt.show()

Results

Input

Output

You can download uncompressed input and output images here:

input
result

Summary

Seems pretty easy, right? With a little amount of work with, I was able to leverage power of SWT algorithm from Python with the speed of low-level C code.

Please comment if you encounter any problem and share if you like it!

Additional links & resources

full project on my github: https://github.com/marrrcin/swt-python
original paper about SWT: https://www.microsoft.com/en-us/research/publication/stroke-width-transform/
dataset of photos with text: http://www.liv.ic.unicamp.br/~minetto/datasets/text/IMAGES/CVPR/ORIG/

Schedule Python Apache Beam DataFlow pipeline using App Engine CRON service

Marcin Zabłocki — Fri, 30 Jun 2017 22:00:00 +0000

Apache Beam is a relatively new framework, which claims to deliver unified, parallel processing model for the data. Apache Beam with Google DataFlow can be used in various data processing scenarios like: ETLs (Extract Transform Load), data migrations and machine learning pipelines. This post explains how to run Apache Beam Python pipeline using Google DataFlow and then how to deploy this pipeline to App Engine in order to run it i.e on a daily basis using App Engine CRON service. There are some tutorials explaining how to create basic pipelines and how to run them using Apache Beam DataFlow Runner, but there is not much information how to deploy those pipelines to be runned without user interaction, which should be the first thing to do, since no one wants to trigger big data pipelines by hand.

As an example pipeline I will show wordcount on a few books of Shakespeare.

TL;DR

Use App Engine CRON to launch Google DataFlow pipeline written in Apache Beam Python SDK. Link to github: https://github.com/marrrcin/python-beam-dataflow-cron

Prerequisites

Python 2.7.12+
Apache Beam SDK for Python 2.0.0
Google Cloud Platform account to run the pipeline in DataFlow and store results in Cloud Storage
Google Cloud SDK for Python
basic knowledge of Apache Beam

The pipeline

The idea behind wordcount pipeline in Apache Beam is simple - actually it’s almost the same as for MapReduce, but simpler in the implementation, because of the Apache Beam capabilities and expressiveness:

read all input files
tokenize the text
for every token, emit tuple (word, 1) - mapping phase
group tuples by word
sum values for every group - aggregation phase
format and save output

Pipeline implementation

Here is python code for the pipeline described above. Comments and code explanation is below.

python

from __future__ import print_function, absolute_import
import apache_beam as beam
from apache_beam.io import ReadFromText
from apache_beam.io import WriteToText
import logging
logging.basicConfig(level=logging.INFO)


class FindWords(beam.DoFn):
    def process(self, element):
        import re as regex
        return regex.findall(r"[A-Za-z\']+", element)


class CountWordsTransform(beam.PTransform):
    def expand(self, p_collection):
        # p_collection is collection of loaded lines
        return (p_collection
                | "Find words" >> (beam.ParDo(FindWords()).with_input_types(unicode))
                | "Pair With One" >> beam.Map(lambda word: (word, 1))
                | "Group By" >> beam.GroupByKey()
                | "Aggregate Groups" >> beam.Map(lambda (word, ones): (word, sum(ones))))


def run():
    import time
    gcs_path = "gs://marcin-playground/dataflow"
    pipeline =  beam.Pipeline(runner="DataflowRunner", argv=[
        "--project", "your-project-name",
        "--staging_location", ("%s/staging_location" % gcs_path),
        "--temp_location", ("%s/temp" % gcs_path),
        "--output", ("%s/output" % gcs_path),
        "--setup_file", "./setup.py"
    ])
    (pipeline
     | "Load" >> ReadFromText("gs://marcin-playground/books/*.txt")
     | "Count Words" >> CountWordsTransform()
     | "Format Output" >> beam.Map(lambda (word, count): "{0}: {1}".format(word, count))
     | "Save" >> WriteToText("{0}/output/wordcount{1}".format(gcs_path, int(time.time())))
     )
    pipeline.run()

Besides the required imports, first important part is the run() method - it’s an entry point to the pipeline. Whan you can observe there is that it does not follow the Apache Beam documentation convention, where beam.Pipeline objects are wrapped inside the python with clause. I did it on purpose - thanks to that, the code will not wait for the pipeline to complete, but rather it will dispatch the job to Google DataFlow and return (see “DataflowRunner” in the constructor). Other configuration parameters are self-descriptive and they are required by the Beam SDK.

First, the pipeline reads all book files (link to those files is at the end of this post) from Google Cloud Storage - you need to create GCS bucket and put the input files into the folders in order to use them in the pipeline.

Count Words is a second step. It uses my implementation of beam.PTransform class, which can be used to join multiple transforms into single high-level piece. The implementation itself consists of 4 steps:

Finding the words - for simplicity, I assume that word only contains lower and upper case letters and/or an apostrophe. In the FindWords class, which is a custom implementation of beam.DoFn, the def process(self, element) function is responsible of transforming each line (element parameter) of the readed books in to something else, in this case - the output will be a list of words.
Next, each list of words is transformed into tuples in form (word, 1), it’s just a matter of invoking beam.Map built-in function
Third step is grouping tuples by word, so we can then aggregate (count) them in total
Last step is to acctualy aggregate them, thanks to the fact that we’ve emmited tuples with 1, we can just sum them to obtain the total number of occurences for each word

Format Output does only one simple thing - transforms each tuple in form (word, number of occurences) to readable string in format: my word: 123.

Last step is to Save results into file. For the case of running the pipeline multiple times, output folder will have current timestamp in the name.

Deploying Python DataFlow pipeline to Google App Engine

Now when the pipeline is ready, we can prepare python web application which will trigger DataFlow jobs. For this task, I will use Flask framework.

Project structure

Start with creating the project structure:

Description of the structure:

app.yaml - definition of Google App Engine application
cron.yaml - definition of Google App Engine CRON job
main.py - entry point for the web application
dataflow_pipeline module - the pipeline described above, wrapped inside the python module
requirements.txt - file for pip package manager, which contains list of all required packages to run the application and the pipeline
appengine_config.py - standard App Engine python configuration file
lib - local folder with all pip-installed packages from requirements.txt file
setup.py - setup file, which is required for DataFlow to be able to run the code from the pipeline - when you use libraries like pandas in your pipeline code, you must notify the DataFlow to install this package on every worker machine that will be running the pipeline - that’s why the setup.py file is required

Configuration

What is really important in the requirements.txt file. For my pipeline, it’s contents are:

python

Flask==0.12.2
apache-beam[gcp]==2.0.0
gunicorn==19.7.1
google-cloud-dataflow==2.0.0

In order to make the App Engine application to work, all of the required packages are needed to be deployed with the application, that’s why you need to install them locally by invoking:

powershell

pip install -r requirements.txt -t lib

Also note that apache-beam package is used with [gcp] feature, which will install the capabilities of running the pipeline in DataFlow.

Configuration file for the application looks like this: app.yaml

html

runtime: python
env: flex

entrypoint: gunicorn -b :$PORT main:app

runtime_config:
  python_version: 2

The file describes that we want to use App Engine Flex environment, with python 2.7 runtime.

IMPORTANT: Apache Beam pipelines will not work on App Engine Standard environment, due to some filestystem I/O operations. You will most likely encounter this exception there:

powershell

IOError: [Errno 30] Read-only file system: '/dev/null'
at <module> (/base/data/home/apps/project/20170626t141416.402244965699518249/lib/dill/dill.py:167)
at <module> (/base/data/home/apps/project/20170626t141416.402244965699518249/lib/dill/__init__.py:27)

The solution is just to use Flex environment.

Flask application for starting DataFlow pipeline

Here is the implementation of Flask application with single endpoint: /start-dataflow.

python

from __future__ import print_function, absolute_import
from flask import Flask, request
import logging
import dataflow_pipeline.wordcountpipeline as pipeline

app = Flask(__name__)


@app.route("/start-dataflow")
def start_dataflow():
    is_cron = request.headers.get('X-Appengine-Cron', False)
    if not is_cron:
        return 'Bad Request', 400

    try:
        pipeline.run()
        return "Pipeline started", 200
    except Exception as e:
        logging.exception(e)
        return "Error: <pre>{}</pre>".format(e), 500


@app.errorhandler(500)
def server_error(e):
    logging.exception('An error occurred during a request.')
    return """
    An internal error occurred: <pre>{}</pre>
    See logs for full stacktrace.
    """.format(e), 500


if __name__ == '__main__':
    app.run(host='127.0.0.1', port=8080, debug=True)

From the code above, the most important part is the def start_dataflow() function. It checks for the presence of X-Appengine-Cron HTTP request header. Thanks to that, the app can know, whether the request came from App Engine CRON service or just from external call. For the debugging purposes, you can comment out this part, but remember to NOT DEPLOY application without this check, because then, any user will be able to trigger your pipeline and potentially increase your billing a lot!

After asserting the presence of HTTP header, Apache Beam pipeline is spawned by calling run() method which was defined in our pipeline above. If everything goes fine, web application will return status code 200. If something will go wrong, you will get status code 500 with stacktrace, which you can use to hunt down the issue.

Defining App Engine CRON service to trigger python DataFlow pipeline

Last thing to do is to define cron.yaml file with CRON job description:

html

cron:
- description: trigger dataflow test
  url: /start-dataflow
  schedule: every 30 mins
  target: default

It’s a simple file, which will send GET request to /start-dataflow endpoint every 30 minutes. Of course you can modify the schedule as needed.

Deployment

Deploy web application by invoking:

powershell

gcloud app deploy app.yaml

Note that for large pipelines with lots of external libraries, deploying App Engine Flex environment application might take a while.

Deploy CRON service by invoking:

powershell

gcloud app deploy cron.yaml

Go to Google Cloud Platform Console (https://console.cloud.google.com) and open App Engine tab. You should see your app there.
Under App Engine tab, open Task queues > Cron Jobs - the CRON service should be visible there. You can trigger it manually by using [Run now] button.

Executing the pipeline

After successful job dispatch, you can open GCP Console and navigate to DataFlow tab to see all of your finished / running jobs:

For each job, you can view it’s exectuion statistics, logs and data flow visualization:

Summary

That’s it! I hope this end-to-end example will help you deploy your jobs to Google DataFlow. Please share or leave comment if you like it!

Pro tips

when you want to use library in Apache Beam pipeline i.e in your beam.DoFn implementation, always use local imports (in the method body)
if you use custom python libraries (installed through pip) you must create setup.py file with all the packages listed there, because Google DataFlow workers are deployed only with limited number of preinstalled packages

Additional links & resources

full project on my github: https://github.com/marrrcin/python-beam-dataflow-cron
Shakespeare’s book collection: http://www.gutenberg.org/ebooks/author/65
Apache Beam Python Guide: https://beam.apache.org/documentation/programming-guide/

Sentiment analysis of tweets with Python, NLTK, word2vec & scikit-learn

Marcin Zabłocki — Thu, 25 May 2017 22:00:00 +0000

This post describes full machine learning pipeline used for sentiment analysis of twitter posts divided by 3 categories: positive, negative and neutral. For this task I used python with: scikit-learn, nltk, pandas, word2vec and xgboost packages.

TL;DR

Detailed description & report of tweets sentiment analysis using machine learning techniques in Python. With details, but this is not a tutorial.

Jupyter Notebook of this post

This post is compiled version of Jupyter Notebook, which you can download here: https://github.com/marrrcin/ml-twitter-sentiment-analysis

Abstract

The goal of this project was to predict sentiment for the given Twitter post using Python. Sentiment analysis can predict many different emotions attached to the text, but in this report only 3 major were considered: positive, negative and neutral. The training dataset was small (just over 5900 examples) and the data within it was highly skewed, which greatly impacted on the difficulty of building good classifier. After creating a lot of custom features, utilizing both bag-of-words and word2vec representations and applying the Extreme Gradient Boosting algorithm, the classification accuracy at level of 58% was achieved.

Used Python Libraries

Data was pre-processed using pandas, gensim and numpy libraries and the learning/validating process was built with scikit-learn. Plots were created using plotly.

python

from collections import Counter
import nltk
import pandas as pd
from emoticons import EmoticonDetector
import re as regex
import numpy as np
import plotly
from plotly import graph_objs
from sklearn.metrics import f1_score, precision_score, recall_score, accuracy_score
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV, RandomizedSearchCV
from time import time
import gensim

# plotly configuration
plotly.offline.init_notebook_mode()

Notebook code convention

This report was first prepared as a classical Python project using object oriented programming with maintainability in mind. In order to show this project as a Jupyter Notebook, the classes had to be splitted into multiple code-cells. In order to do so, the classes are suffixed with _PurposeOfThisSnippet name and they inherit one from another. The final class will be then run and the results will be shown.

Data source

The input data consisted two CSV files: train.csv (5971 tweets) and test.csv (4000 tweets) - one for training and one for testing. Format of the data was the following (test data didn’t contain Category column):

Id	Category	Tweet
635930169241374720	neutral	IOS 9 App Transport Security. Mm need to check if my 3rd party network pod supports it

All tweets are in english, so it simplifies the processing and analysis.

Data preprocessing

Loading the data

python

class TwitterData_Initialize():
    data = []
    processed_data = []
    wordlist = []

    data_model = None
    data_labels = None
    is_testing = False
    
    def initialize(self, csv_file, is_testing_set=False, from_cached=None):
        if from_cached is not None:
            self.data_model = pd.read_csv(from_cached)
            return

        self.is_testing = is_testing_set

        if not is_testing_set:
            self.data = pd.read_csv(csv_file, header=0, names=["id", "emotion", "text"])
            self.data = self.data[self.data["emotion"].isin(["positive", "negative", "neutral"])]
        else:
            self.data = pd.read_csv(csv_file, header=0, names=["id", "text"],dtype={"id":"int64","text":"str"},nrows=4000)
            not_null_text = 1 ^ pd.isnull(self.data["text"])
            not_null_id = 1 ^ pd.isnull(self.data["id"])
            self.data = self.data.loc[not_null_id & not_null_text, :]

        self.processed_data = self.data
        self.wordlist = []
        self.data_model = None
        self.data_labels = None

The code snippet above is prepared, to load the data form the given file for further processing, or just read already preprocessed file from the cache. There’s also a distinction between processing testing and training data. As the test.csv file was full of empty entries, they were removed. Additional class properties such as data_model, wordlist etc. will be used further.

python

data = TwitterData_Initialize()
data.initialize("data\\train.csv")
data.processed_data.head(5)

	id	emotion	text
0	635769805279248384	negative	Not Available
1	635930169241374720	neutral	IOS 9 App Transport Security. Mm need to check...
2	635950258682523648	neutral	Mar if you have an iOS device, you should down...
3	636030803433009153	negative	@jimmie_vanagon my phone does not run on lates...
4	636100906224848896	positive	Not sure how to start your publication on iOS?...

Data distribution

First thing that can be done as soon as the data is loaded is to see the data distribution. The training set had the following distribution:

python

df = data.processed_data
neg = len(df[df["emotion"] == "negative"])
pos = len(df[df["emotion"] == "positive"])
neu = len(df[df["emotion"] == "neutral"])
dist = [
    graph_objs.Bar(
        x=["negative","neutral","positive"],
        y=[neg, neu, pos],
)]
plotly.offline.iplot({"data":dist, "layout":graph_objs.Layout(title="Sentiment type distribution in training set")})

Preprocessing steps

The targed of the following preprocessing is to create a Bag-of-Words representation of the data. The steps will execute as follows:

Cleansing
1. Remove URLs
2. Remove usernames (mentions)
3. Remove tweets with Not Available text
4. Remove special characters
5. Remove numbers
Text processing
1. Tokenize
2. Transform to lowercase
3. Stem
Build word list for Bag-of-Words

Cleansing

For the purpose of cleansing, the TwitterCleanup class was created. It consists methods allowing to execute all of the tasks show in the list above. Most of those is done using regular expressions. The class exposes it’s interface through iterate() method - it yields every cleanup method in proper order.

python

class TwitterCleanuper:
    def iterate(self):
        for cleanup_method in [self.remove_urls,
                               self.remove_usernames,
                               self.remove_na,
                               self.remove_special_chars,
                               self.remove_numbers]:
            yield cleanup_method

    @staticmethod
    def remove_by_regex(tweets, regexp):
        tweets.loc[:, "text"].replace(regexp, "", inplace=True)
        return tweets

    def remove_urls(self, tweets):
        return TwitterCleanuper.remove_by_regex(tweets, regex.compile(r"http.?://[^\s]+[\s]?"))

    def remove_na(self, tweets):
        return tweets[tweets["text"] != "Not Available"]

    def remove_special_chars(self, tweets):  # it unrolls the hashtags to normal words
        for remove in map(lambda r: regex.compile(regex.escape(r)), [",", ":", "\"", "=", "&", ";", "%", "$",
                                                                     "@", "%", "^", "*", "(", ")", "{", "}",
                                                                     "[", "]", "|", "/", "\\", ">", "<", "-",
                                                                     "!", "?", ".", "'",
                                                                     "--", "---", "#"]):
            tweets.loc[:, "text"].replace(remove, "", inplace=True)
        return tweets

    def remove_usernames(self, tweets):
        return TwitterCleanuper.remove_by_regex(tweets, regex.compile(r"@[^\s]+[\s]?"))

    def remove_numbers(self, tweets):
        return TwitterCleanuper.remove_by_regex(tweets, regex.compile(r"\s?[0-9]+\.?[0-9]*"))

The loaded tweets can be now cleaned.

python

class TwitterData_Cleansing(TwitterData_Initialize):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        
    def cleanup(self, cleanuper):
        t = self.processed_data
        for cleanup_method in cleanuper.iterate():
            if not self.is_testing:
                t = cleanup_method(t)
            else:
                if cleanup_method.__name__ != "remove_na":
                    t = cleanup_method(t)

        self.processed_data = t

python

data = TwitterData_Cleansing(data)
data.cleanup(TwitterCleanuper())
data.processed_data.head(5)

	id	emotion	text
1	635930169241374720	neutral	IOS App Transport Security Mm need to check if...
2	635950258682523648	neutral	Mar if you have an iOS device you should downl...
3	636030803433009153	negative	my phone does not run on latest IOS which may ...
4	636100906224848896	positive	Not sure how to start your publication on iOS ...
5	636176272947744772	neutral	Two Dollar Tuesday is here with Forklift Quick...

Tokenization & stemming

For the text processing, nltk library is used. First, the tweets are tokenized using nlkt.word_tokenize and then, stemming is done using PorterStemmer as the tweets are 100% in english.

python

class TwitterData_TokenStem(TwitterData_Cleansing):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        
    def stem(self, stemmer=nltk.PorterStemmer()):
        def stem_and_join(row):
            row["text"] = list(map(lambda str: stemmer.stem(str.lower()), row["text"]))
            return row

        self.processed_data = self.processed_data.apply(stem_and_join, axis=1)

    def tokenize(self, tokenizer=nltk.word_tokenize):
        def tokenize_row(row):
            row["text"] = tokenizer(row["text"])
            row["tokenized_text"] = [] + row["text"]
            return row

        self.processed_data = self.processed_data.apply(tokenize_row, axis=1)

python

data = TwitterData_TokenStem(data)
data.tokenize()
data.stem()
data.processed_data.head(5)

	id	emotion	text	tokenized_text
1	635930169241374720	neutral	[io, app, transport, secur, mm, need, to, chec...	[IOS, App, Transport, Security, Mm, need, to, ...
2	635950258682523648	neutral	[mar, if, you, have, an, io, devic, you, shoul...	[Mar, if, you, have, an, iOS, device, you, sho...
3	636030803433009153	negative	[my, phone, doe, not, run, on, latest, io, whi...	[my, phone, does, not, run, on, latest, IOS, w...
4	636100906224848896	positive	[not, sure, how, to, start, your, public, on, ...	[Not, sure, how, to, start, your, publication,...
5	636176272947744772	neutral	[two, dollar, tuesday, is, here, with, forklif...	[Two, Dollar, Tuesday, is, here, with, Forklif...

Building the wordlist

The wordlist (dictionary) is build by simple count of occurences of every unique word across all of the training dataset.

Before building the final wordlist for the model, let’s take a look at the non-filtered version:

python

words = Counter()
for idx in data.processed_data.index:
    words.update(data.processed_data.loc[idx, "text"])

words.most_common(5)

# output:
# [('the', 3744), ('to', 2477), ('i', 1667), ('a', 1620), ('on', 1557)]

The most commont words (as expected) are the typical english stopwords. We will filter them out, however, as purpose of this analysis is to determine sentiment, words like “not” and “n’t” can influence it greatly. Having this in mind, this word will be whitelisted.

python

stopwords=nltk.corpus.stopwords.words("english")
whitelist = ["n't", "not"]
for idx, stop_word in enumerate(stopwords):
    if stop_word not in whitelist:
        del words[stop_word]
words.most_common(5)

# output
# [('may', 1027), ('tomorrow', 764), ('day', 526), ('go', 499), ('thi', 495)]

Still, there are some words that seem too be occuring to many times, let’s filter them. After some analysis, the lower bound was set to 3.

The wordlist is also saved to the csv file, so the same words can be used for the testing set.

python

class TwitterData_Wordlist(TwitterData_TokenStem):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        
    whitelist = ["n't","not"]
    wordlist = []
        
    def build_wordlist(self, min_occurrences=3, max_occurences=500, stopwords=nltk.corpus.stopwords.words("english"),
                       whitelist=None):
        self.wordlist = []
        whitelist = self.whitelist if whitelist is None else whitelist
        import os
        if os.path.isfile("data\\wordlist.csv"):
            word_df = pd.read_csv("data\\wordlist.csv")
            word_df = word_df[word_df["occurrences"] > min_occurrences]
            self.wordlist = list(word_df.loc[:, "word"])
            return

        words = Counter()
        for idx in self.processed_data.index:
            words.update(self.processed_data.loc[idx, "text"])

        for idx, stop_word in enumerate(stopwords):
            if stop_word not in whitelist:
                del words[stop_word]

        word_df = pd.DataFrame(data={"word": [k for k, v in words.most_common() if min_occurrences < v < max_occurences],
                                     "occurrences": [v for k, v in words.most_common() if min_occurrences < v < max_occurences]},
                               columns=["word", "occurrences"])

        word_df.to_csv("data\\wordlist.csv", index_label="idx")
        self.wordlist = [k for k, v in words.most_common() if min_occurrences < v < max_occurences]

python

data = TwitterData_Wordlist(data)
data.build_wordlist()

python

words = pd.read_csv("data\\wordlist.csv")
x_words = list(words.loc[0:10,"word"])
x_words.reverse()
y_occ = list(words.loc[0:10,"occurrences"])
y_occ.reverse()

dist = [
    graph_objs.Bar(
        x=y_occ,
        y=x_words,
        orientation="h"
)]
plotly.offline.iplot({"data":dist, "layout":graph_objs.Layout(title="Top words in built wordlist")})

Bag-of-words

The data is ready to transform it to bag-of-words representation.

python

class TwitterData_BagOfWords(TwitterData_Wordlist):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        self.wordlist = previous.wordlist
    
    def build_data_model(self):
        label_column = []
        if not self.is_testing:
            label_column = ["label"]

        columns = label_column + list(
            map(lambda w: w + "_bow",self.wordlist))
        labels = []
        rows = []
        for idx in self.processed_data.index:
            current_row = []

            if not self.is_testing:
                # add label
                current_label = self.processed_data.loc[idx, "emotion"]
                labels.append(current_label)
                current_row.append(current_label)

            # add bag-of-words
            tokens = set(self.processed_data.loc[idx, "text"])
            for _, word in enumerate(self.wordlist):
                current_row.append(1 if word in tokens else 0)

            rows.append(current_row)

        self.data_model = pd.DataFrame(rows, columns=columns)
        self.data_labels = pd.Series(labels)
        return self.data_model, self.data_labels

Let’s take a look at the data and see, which words are the most common for particular sentiments.

python

data = TwitterData_BagOfWords(data)
bow, labels = data.build_data_model()
bow.head(5)

	label	wa_bow	not_bow	...
0	neutral	0	0	...
1	neutral	0	0	...
2	negative	1	1	...
3	positive	0	1	...
4	neutral	0	0	...

5 rows × 2185 columns

python

grouped = bow.groupby(["label"]).sum()
words_to_visualize = []
sentiments = ["positive","negative","neutral"]
#get the most 7 common words for every sentiment
for sentiment in sentiments:
    words = grouped.loc[sentiment,:]
    words.sort_values(inplace=True,ascending=False)
    for w in words.index[:7]:
        if w not in words_to_visualize:
            words_to_visualize.append(w)
            
            
#visualize it
plot_data = []
for sentiment in sentiments:
    plot_data.append(graph_objs.Bar(
            x = [w.split("_")[0] for w in words_to_visualize],
            y = [grouped.loc[sentiment,w] for w in words_to_visualize],
            name = sentiment
    ))
    
plotly.offline.iplot({
        "data":plot_data,
        "layout":graph_objs.Layout(title="Most common words across sentiments")
    })

Some of the most common words show high distinction between classes like go and see and other are occuring in similiar amount for every class (plan, obama).

None of the most common words is unique to the negative class. At this point, it’s clear that skewed data distribution will be a problem in distinguishing negative tweets from the others.

Classification

First of all, lets establish seed for random numbers generators.

python

import random
seed = 666
random.seed(seed)

The following utility function will train the classifier and show the F1, precision, recall and accuracy scores.

python

def test_classifier(X_train, y_train, X_test, y_test, classifier):
    log("")
    log("===============================================")
    classifier_name = str(type(classifier).__name__)
    log("Testing " + classifier_name)
    now = time()
    list_of_labels = sorted(list(set(y_train)))
    model = classifier.fit(X_train, y_train)
    log("Learing time {0}s".format(time() - now))
    now = time()
    predictions = model.predict(X_test)
    log("Predicting time {0}s".format(time() - now))

    precision = precision_score(y_test, predictions, average=None, pos_label=None, labels=list_of_labels)
    recall = recall_score(y_test, predictions, average=None, pos_label=None, labels=list_of_labels)
    accuracy = accuracy_score(y_test, predictions)
    f1 = f1_score(y_test, predictions, average=None, pos_label=None, labels=list_of_labels)
    log("=================== Results ===================")
    log("            Negative     Neutral     Positive")
    log("F1       " + str(f1))
    log("Precision" + str(precision))
    log("Recall   " + str(recall))
    log("Accuracy " + str(accuracy))
    log("===============================================")

    return precision, recall, accuracy, f1

def log(x):
    #can be used to write to log file
    print(x)

Experiment 1: BOW + Naive Bayes

It is nice to see what kind of results we might get from such simple model. The bag-of-words representation is binary, so Naive Bayes Classifier seems like a nice algorithm to start the experiments.

The experiment will be based on 7:3 train:test stratified split.

python

from sklearn.naive_bayes import BernoulliNB
X_train, X_test, y_train, y_test = train_test_split(bow.iloc[:, 1:], bow.iloc[:, 0],
                                                    train_size=0.7, stratify=bow.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, BernoulliNB())

markdown

    ===============================================
    Testing BernoulliNB
    Learing time 0.451092004776001s
    Predicting time 0.1190040111541748s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.38949672  0.45072993  0.71369782]
    Precision[ 0.45408163  0.48431373  0.65906623]
    Recall   [ 0.34099617  0.42150171  0.77820513]
    Accuracy 0.579594345421
    ===============================================

Result with accuracy at level of 58% seems to be quite nice result for such basic algorithm like Naive Bayes (having in mind that random classifier would yield result of around 33% accuracy). This performance may not hold for the final testing set. In order to see how the NaiveBayes performs in more general cases, 8-fold crossvalidation is used. The 8 fold is used, to optimize speed of testing on my 8-core machine.

python

def cv(classifier, X_train, y_train):
    log("===============================================")
    classifier_name = str(type(classifier).__name__)
    now = time()
    log("Crossvalidating " + classifier_name + "...")
    accuracy = [cross_val_score(classifier, X_train, y_train, cv=8, n_jobs=-1)]
    log("Crosvalidation completed in {0}s".format(time() - now))
    log("Accuracy: " + str(accuracy[0]))
    log("Average accuracy: " + str(np.array(accuracy[0]).mean()))
    log("===============================================")
    return accuracy

python

nb_acc = cv(BernoulliNB(), bow.iloc[:,1:], bow.iloc[:,0])

markdown

    ===============================================
    Crossvalidating BernoulliNB...
    Crosvalidation completed in 4.4375975131988525s
    Accuracy: [ 0.54639175  0.48820059  0.28023599  0.31415929  0.32743363  0.50073855
      0.47119645  0.53106509]
    Average accuracy: 0.432427668406
    ===============================================

This result no longer looks optimistic. For some of the splits, Naive Bayes classifier showed performance below the performance of random classifier.

Additional features

In order to not push any other aglorithm to the limit on the current data model, let’s try to add some features that might help to classify tweets.

A common sense suggest that special characters like exclamation marks and the casing might be important in the task of determining the sentiment. The following features will be added to the data model:

Feature name	Explanation
Number of uppercase	people tend to express with either positive or negative emotions by using A LOT OF UPPERCASE WORDS
Number of !	exclamation marks are likely to increase the strength of opinion
Number of ?	might distinguish neutral tweets - seeking for information
Number of positive emoticons	positive emoji will most likely not occur in the negative tweets
Number of negative emoticons	inverse to the one above
Number of …	commonly used in commenting something
Number of quotations	same as above
Number of mentions	sometimes people put a lot of mentions on positive tweets, to share something good
Number of hashtags	just for the experiment
Number of urls	similiar to the number of mentions

Extraction of those features must be done before any preprocessing happens.

For the purpose of emoticons, the EmoticonDetector class is created. The file emoticons.txt contains list of positive and negative emoticons, which are used.

python

class EmoticonDetector:
    emoticons = {}

    def __init__(self, emoticon_file="data\\emoticons.txt"):
        from pathlib import Path
        content = Path(emoticon_file).read_text()
        positive = True
        for line in content.split("\n"):
            if "positive" in line.lower():
                positive = True
                continue
            elif "negative" in line.lower():
                positive = False
                continue

            self.emoticons[line] = positive

    def is_positive(self, emoticon):
        if emoticon in self.emoticons:
            return self.emoticons[emoticon]
        return False

    def is_emoticon(self, to_check):
        return to_check in self.emoticons

python

class TwitterData_ExtraFeatures(TwitterData_Wordlist):
    def __init__(self):
        pass
    
    def build_data_model(self):
        extra_columns = [col for col in self.processed_data.columns if col.startswith("number_of")]
        label_column = []
        if not self.is_testing:
            label_column = ["label"]

        columns = label_column + extra_columns + list(
            map(lambda w: w + "_bow",self.wordlist))
        
        labels = []
        rows = []
        for idx in self.processed_data.index:
            current_row = []

            if not self.is_testing:
                # add label
                current_label = self.processed_data.loc[idx, "emotion"]
                labels.append(current_label)
                current_row.append(current_label)

            for _, col in enumerate(extra_columns):
                current_row.append(self.processed_data.loc[idx, col])

            # add bag-of-words
            tokens = set(self.processed_data.loc[idx, "text"])
            for _, word in enumerate(self.wordlist):
                current_row.append(1 if word in tokens else 0)

            rows.append(current_row)

        self.data_model = pd.DataFrame(rows, columns=columns)
        self.data_labels = pd.Series(labels)
        return self.data_model, self.data_labels
    
    def build_features(self):
        def count_by_lambda(expression, word_array):
            return len(list(filter(expression, word_array)))

        def count_occurences(character, word_array):
            counter = 0
            for j, word in enumerate(word_array):
                for char in word:
                    if char == character:
                        counter += 1

            return counter

        def count_by_regex(regex, plain_text):
            return len(regex.findall(plain_text))

        self.add_column("splitted_text", map(lambda txt: txt.split(" "), self.processed_data["text"]))

        # number of uppercase words
        uppercase = list(map(lambda txt: count_by_lambda(lambda word: word == word.upper(), txt),
                             self.processed_data["splitted_text"]))
        self.add_column("number_of_uppercase", uppercase)

        # number of !
        exclamations = list(map(lambda txt: count_occurences("!", txt),
                                self.processed_data["splitted_text"]))

        self.add_column("number_of_exclamation", exclamations)

        # number of ?
        questions = list(map(lambda txt: count_occurences("?", txt),
                             self.processed_data["splitted_text"]))

        self.add_column("number_of_question", questions)

        # number of ...
        ellipsis = list(map(lambda txt: count_by_regex(regex.compile(r"\.\s?\.\s?\."), txt),
                            self.processed_data["text"]))

        self.add_column("number_of_ellipsis", ellipsis)

        # number of hashtags
        hashtags = list(map(lambda txt: count_occurences("#", txt),
                            self.processed_data["splitted_text"]))

        self.add_column("number_of_hashtags", hashtags)

        # number of mentions
        mentions = list(map(lambda txt: count_occurences("@", txt),
                            self.processed_data["splitted_text"]))

        self.add_column("number_of_mentions", mentions)

        # number of quotes
        quotes = list(map(lambda plain_text: int(count_occurences("'", [plain_text.strip("'").strip('"')]) / 2 +
                                                 count_occurences('"', [plain_text.strip("'").strip('"')]) / 2),
                          self.processed_data["text"]))

        self.add_column("number_of_quotes", quotes)

        # number of urls
        urls = list(map(lambda txt: count_by_regex(regex.compile(r"http.?://[^\s]+[\s]?"), txt),
                        self.processed_data["text"]))

        self.add_column("number_of_urls", urls)

        # number of positive emoticons
        ed = EmoticonDetector()
        positive_emo = list(
            map(lambda txt: count_by_lambda(lambda word: ed.is_emoticon(word) and ed.is_positive(word), txt),
                self.processed_data["splitted_text"]))

        self.add_column("number_of_positive_emo", positive_emo)

        # number of negative emoticons
        negative_emo = list(map(
            lambda txt: count_by_lambda(lambda word: ed.is_emoticon(word) and not ed.is_positive(word), txt),
            self.processed_data["splitted_text"]))

        self.add_column("number_of_negative_emo", negative_emo)
        
    def add_column(self, column_name, column_content):
        self.processed_data.loc[:, column_name] = pd.Series(column_content, index=self.processed_data.index)

python

data = TwitterData_ExtraFeatures()
data.initialize("data\\train.csv")
data.build_features()
data.cleanup(TwitterCleanuper())
data.tokenize()
data.stem()
data.build_wordlist()
data_model, labels = data.build_data_model()

Logic behind extra features

Let’s see how (some) of the extra features separate the data set. Some of them, i.e number exclamation marks, number of pos/neg emoticons do this really well. Despite of the good separation, those features sometimes occur only on small subset of the training dataset.

python

sentiments = ["positive","negative","neutral"]
plots_data_ef = []
for what in map(lambda o: "number_of_"+o,["positive_emo","negative_emo","exclamation","hashtags","question"]):
    ef_grouped = data_model[data_model[what]>=1].groupby(["label"]).count()
    plots_data_ef.append({"data":[graph_objs.Bar(
            x = sentiments,
            y = [ef_grouped.loc[s,:][0] for s in sentiments],
    )], "title":"How feature \""+what+"\" separates the tweets"})
    

for plot_data_ef in plots_data_ef:
    plotly.offline.iplot({
            "data":plot_data_ef["data"],
            "layout":graph_objs.Layout(title=plot_data_ef["title"])
    })

Experiment 2: extended features + Random Forest

As a second attempt on the classification the Random Forest will be used.

python

from sklearn.ensemble import RandomForestClassifier
X_train, X_test, y_train, y_test = train_test_split(data_model.iloc[:, 1:], data_model.iloc[:, 0],
                                                    train_size=0.7, stratify=data_model.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, RandomForestClassifier(random_state=seed,n_estimators=403,n_jobs=-1))

markdown

    ===============================================
    Testing RandomForestClassifier
    Learing time 6.9144287109375s
    Predicting time 0.21802711486816406s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.24501425  0.47944007  0.70340909]
    Precision[ 0.47777778  0.49192101  0.63163265]
    Recall   [ 0.16475096  0.46757679  0.79358974]
    Accuracy 0.575291948371
    ===============================================

The accuracy for the initial split was lower than the one for the Naive Bayes, but let’s see what happens during crossvalidation:

python

rf_acc = cv(RandomForestClassifier(n_estimators=403,n_jobs=-1, random_state=seed),data_model.iloc[:, 1:], data_model.iloc[:, 0])

markdown

    ===============================================
    Crossvalidating RandomForestClassifier...
    Crosvalidation completed in 70.09595036506653s
    Accuracy: [ 0.54344624  0.50442478  0.37905605  0.27876106  0.37905605  0.52141802
      0.5155096   0.55621302]
    Average accuracy: 0.459735602399
    ===============================================

It looks better, however it’s still not much above accuracy of the random classifier and barely better than Naive Bayes classifier.

We can observe a low recall level of the RandomForest classifier for the negative class, which may be caused by the data skewness.

More features - word2vec

The overall performance of the previous classifiers could be enhanced by performing time-consuming parameters adjustments, however there’s not guarantee on how big the gain will be.

If the out-of-the-shelf methods did not performed well, it seems that there’s not much in the data itself. The next idea to add more into data model is to use word2vec representation of a tweet to perform classification.

The word2vec allows to transform words into vectors of numbers. Those vectors represent abstract features, that describe the word similarities and relationships (i.e co-occurence).

What is the best in the word2vec is that operations on the vectors approximately keep the characteristics of the words, so that joining (averaging) vectors from the words from sentence procude vector that is likely to represent the general topic of the sentence.

A lot of pre-trained word2vec models exists, and some of them were trained on huge volumes of data. For the purpose of this analysis, the one trained on over 2 billion of tweets with 200 dimensions (one vector consists of 200 numbers) is used. The pre-trained model can be downloaded here: https://github.com/3Top/word2vec-api

From GloVe to word2vec

In order to use GloVe-trained model in gensim library, it needs to be converted to word2vec format. The only difference between those formats is that word2vec text files starts with two numbers: number of lines in file and number of dimensions. The file glove.twitter.27B.200d.txt does not contain those lines.

Unfortunaltely, this text file size is over 1.9GB and text editors cannot be used to open and modify it in reasonable amount of time, this C# snippet adds this required line (sorry that it’s not Python, but I was having memory problems with encoding of the file in Python. It’s required to use x64 target):

csharp

using (var fileStream = new FileStream("glove.twitter.27B.200d.txt", FileMode.Open,FileAccess.ReadWrite))
{
    var lines = new LinkedList<string>();
    using (var streamReader = new StreamReader(fileStream))
    {
        while (!streamReader.EndOfStream)
        {
            lines.AddLast(streamReader.ReadLine());
        }
    }
    lines.AddFirst("1193514 200");
    File.WriteAllLines("word2vec.twitter.27B.200d.txt.txt", lines);
}

### Already modified GloVe for file The file that has the first line appended can be downloaded from here (622MB 7-zip file, ultra compression): https://marcin.egnyte.com/dl/KiIlEYaFiU/word2vec.twitter.27B.200d.7z_

Using Word2Vec

The following class exposes a easy to use interface over the word2vec API from gensim library:

python

class Word2VecProvider(object):
    word2vec = None
    dimensions = 0

    def load(self, path_to_word2vec):
        self.word2vec = gensim.models.Word2Vec.load_word2vec_format(path_to_word2vec, binary=False)
        self.word2vec.init_sims(replace=True)
        self.dimensions = self.word2vec.vector_size

    def get_vector(self, word):
        if word not in self.word2vec.vocab:
            return None

        return self.word2vec.syn0norm[self.word2vec.vocab[word].index]

    def get_similarity(self, word1, word2):
        if word1 not in self.word2vec.vocab or word2 not in self.word2vec.vocab:
            return None

        return self.word2vec.similarity(word1, word2)

python

word2vec = Word2VecProvider()

# REPLACE PATH TO THE FILE
word2vec.load("C:\\__\\machinelearning\\glove.twitter.27B.200d.txt")

Extra features from word2vec

Besides the 200 additional features from the word2vec representation, I had an idea of 3 more features. If word2vec allows to find similarity between words, that means it can find similarity to the specific emotion-representing words. The first idea was to compute similarity of the whole tweet with words from labels: positive, negative, neutral. Since the purpose was to find the sentiment, I thought that it will be better to find similarity with more expressive words such as: good and bad. For the neutral sentiment, I’ve used word information, since most of the tweets with neutral sentiment were giving the information.

The features were builded by computing mean similarity of the whole tweet to the given word. Then, those mean values were normalized to [0;1] in order to deal with different word count across tweets.

Final data model

The final data model will contain:

extra text features (number of: !, ?, :-) etc)
word2vec similarity to “good”, “bad” and “information” words
word2vec 200 dimension averaged representation of a tweet
bag-of-word representation of a tweet

python

class TwitterData(TwitterData_ExtraFeatures):
    
    def build_final_model(self, word2vec_provider, stopwords=nltk.corpus.stopwords.words("english")):
        whitelist = self.whitelist
        stopwords = list(filter(lambda sw: sw not in whitelist, stopwords))
        extra_columns = [col for col in self.processed_data.columns if col.startswith("number_of")]
        similarity_columns = ["bad_similarity", "good_similarity", "information_similarity"]
        label_column = []
        if not self.is_testing:
            label_column = ["label"]

        columns = label_column + ["original_id"] + extra_columns + similarity_columns + list(
            map(lambda i: "word2vec_{0}".format(i), range(0, word2vec_provider.dimensions))) + list(
            map(lambda w: w + "_bow",self.wordlist))
        labels = []
        rows = []
        for idx in self.processed_data.index:
            current_row = []

            if not self.is_testing:
                # add label
                current_label = self.processed_data.loc[idx, "emotion"]
                labels.append(current_label)
                current_row.append(current_label)

            current_row.append(self.processed_data.loc[idx, "id"])

            for _, col in enumerate(extra_columns):
                current_row.append(self.processed_data.loc[idx, col])

            # average similarities with words
            tokens = self.processed_data.loc[idx, "tokenized_text"]
            for main_word in map(lambda w: w.split("_")[0], similarity_columns):
                current_similarities = [abs(sim) for sim in
                                        map(lambda word: word2vec_provider.get_similarity(main_word, word.lower()), tokens) if
                                        sim is not None]
                if len(current_similarities) <= 1:
                    current_row.append(0 if len(current_similarities) == 0 else current_similarities[0])
                    continue
                max_sim = max(current_similarities)
                min_sim = min(current_similarities)
                current_similarities = [((sim - min_sim) / (max_sim - min_sim)) for sim in
                                        current_similarities]  # normalize to <0;1>
                current_row.append(np.array(current_similarities).mean())

            # add word2vec vector
            tokens = self.processed_data.loc[idx, "tokenized_text"]
            current_word2vec = []
            for _, word in enumerate(tokens):
                vec = word2vec_provider.get_vector(word.lower())
                if vec is not None:
                    current_word2vec.append(vec)

            averaged_word2vec = list(np.array(current_word2vec).mean(axis=0))
            current_row += averaged_word2vec

            # add bag-of-words
            tokens = set(self.processed_data.loc[idx, "text"])
            for _, word in enumerate(self.wordlist):
                current_row.append(1 if word in tokens else 0)

            rows.append(current_row)

        self.data_model = pd.DataFrame(rows, columns=columns)
        self.data_labels = pd.Series(labels)
        return self.data_model, self.data_labels

python

td = TwitterData()
td.initialize("data\\train.csv")
td.build_features()
td.cleanup(TwitterCleanuper())
td.tokenize()
td.stem()
td.build_wordlist()
td.build_final_model(word2vec)

td.data_model.head(5)

	label	original_id	number_of_uppercase	number_of_question	number_of_mentions	number_of_urls	...
0	neutral	635930169241374720	2	0	0	1	...
1	neutral	635950258682523648	0	0	0	1	...
2	negative	636030803433009153	2	0	1	0	...
3	positive	636100906224848896	0	1	0	1	...
4	neutral	636176272947744772	4	0	0	1	...

5 rows × 2399 columns

python

data_model = td.data_model
data_model.drop("original_id",axis=1,inplace=True)

Logic behind word2vec extra features

In order to show, why the 3 extra features built using word2vec might help in sentiment analysis. The chart below shows how many tweets from given category were dominating (had highest value) on similarity to those words. Although the words doesn’t seem to separate the sentiments themselves, the differences between them in addition to other parameters, may help the classification process - i.e when tweet has highest value on good_similarity it’s more likely for it to be classified to have positive sentiment.

python

columns_to_plot = ["bad_similarity", "good_similarity", "information_similarity"]
bad, good, info = columns_to_plot
sentiments = ["positive","negative","neutral"]
only_positive = data_model[data_model[good]>=data_model[bad]]
only_positive = only_positive[only_positive[good]>=only_positive[info]].groupby(["label"]).count()

only_negative = data_model[data_model[bad] >= data_model[good]]
only_negative = only_negative[only_negative[bad] >= only_negative[info]].groupby(["label"]).count()

only_info = data_model[data_model[info]>=data_model[good]]
only_info = only_info[only_info[info]>=only_info[bad]].groupby(["label"]).count()

plot_data_w2v = []
for sentiment in sentiments:
    plot_data_w2v.append(graph_objs.Bar(
            x = ["good","bad", "information"],
            y = [only_positive.loc[sentiment,:][0], only_negative.loc[sentiment,:][0], only_info.loc[sentiment,:][0]],
            name = "Number of dominating " + sentiment
    ))
    
plotly.offline.iplot({
        "data":plot_data_w2v,
        "layout":graph_objs.Layout(title="Number of tweets dominating on similarity to: good, bad, information")
    })

Experiment 3: full model + Random Forest

The model is now complete. With a lot of new features, the learning algorithms should perform totally differently on the new data set.

python

X_train, X_test, y_train, y_test = train_test_split(data_model.iloc[:, 1:], data_model.iloc[:, 0],
                                                    train_size=0.7, stratify=data_model.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, RandomForestClassifier(n_estimators=403,n_jobs=-1, random_state=seed))
rf_acc = cv(RandomForestClassifier(n_estimators=403,n_jobs=-1,random_state=seed),data_model.iloc[:, 1:], data_model.iloc[:, 0])

markdown

    ===============================================
    Testing RandomForestClassifier
    Learing time 3.579137086868286s
    Predicting time 0.2350320816040039s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.08695652  0.48138056  0.72029835]
    Precision[ 0.8         0.51456311  0.61622607]
    Recall   [ 0.04597701  0.45221843  0.86666667]
    Accuracy 0.585740626921
    ===============================================
    ===============================================
    Crossvalidating RandomForestClassifier...
    Crosvalidation completed in 41.57810711860657s
    Accuracy: [ 0.58468336  0.52654867  0.54129794  0.54867257  0.51622419  0.5465288
      0.58493353  0.56508876]
    Average accuracy: 0.551747226492
    ===============================================

As we can see, the average accuracy from crossvalidation is almost 58%, and the results from the crossvalidation runs are more stable - they never drop below 51%. It might seem that the algorithm will perform pretty well on the testing set. There’s a one problem with that - recall for the 7:3 split shows, that only about 3% of the negative tweets from the whole set were recognized properly - it seems like the algorithm is good with recognition of postive vs. neutral cases, but it does really poor job when it comes to recognize negative sentiment. Let’s try something else.

Experiment 4: full model + XGBoost

XGBoost is relatively new machine learning algorithm based on decision trees and boosting. It is highly scalable and provides results, which are often higher than those obtained using popular algorithms such as Random Forest or SVM.

Important: XGBoost exposes scikit-learn interface, but it needs to be installed as an additional python package. See this page to see more: https://xgboost.readthedocs.io/en/latest/build.html

python

from xgboost import XGBClassifier as XGBoostClassifier

python


X_train, X_test, y_train, y_test = train_test_split(data_model.iloc[:, 1:], data_model.iloc[:, 0],
                                                    train_size=0.7, stratify=data_model.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, XGBoostClassifier(seed=seed))

markdown

    ===============================================
    Testing XGBClassifier
    Learing time 94.12349128723145s
    Predicting time 0.2379899024963379s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.33595801  0.50823938  0.72674419]
    Precision[ 0.53333333  0.51675485  0.66489362]
    Recall   [ 0.24521073  0.5         0.80128205]
    Accuracy 0.60356484327
    ===============================================

60% accuracy is the highest result yet. Still, it’s only a result on the splitted training set, let’s see what happens when we crossvalidate this algorithm (with default parameters).

python

xgb_acc = cv(XGBoostClassifier(seed=seed),data_model.iloc[:, 1:], data_model.iloc[:, 0])

markdown

    ===============================================
    Crossvalidating XGBClassifier...
    Crosvalidation completed in 335.4184989929199s
    Accuracy: [ 0.62592047  0.53834808  0.50737463  0.47492625  0.42182891  0.53471196
      0.56277696  0.54585799]
    Average accuracy: 0.526468157158
    ===============================================

Averaged accuracy seems to be lower and less stable than for the Random Forest, but given over 8 times better recall for the negative cases, the XGBoost seems to be a good starting point for the final classifier.

Finding best parameters for XGBoost

Warning the code bellow executes for a really long time. On computer with i7@3.4GHz processor it took over 90 minutes to perform 5 iterations of RandomizedSearchCV, so depending on your resources, you can find even better parameters.

The following utility functions are used to find the best parameters for the XGBoost using RandomizedSearchCV funciton.

python

def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidates = np.flatnonzero(results['rank_test_score'] == i)
        for candidate in candidates:
            log("Model with rank: {0}".format(i))
            log("Mean validation score: {0:.3f} (std: {1:.3f})".format(
                results['mean_test_score'][candidate],
                results['std_test_score'][candidate]))
            log("Parameters: {0}".format(results['params'][candidate]))
            log("")

def best_fit(X_train, y_train, n_iter=5):
    
    parameters = {
        "n_estimators":[103,201, 403],
        "max_depth":[3,10,15, 30],
        "objective":["multi:softmax","binary:logistic"],
        "learning_rate":[0.05, 0.1, 0.15, 0.3]
    }

    rand_search = RandomizedSearchCV(XGBoostClassifier(seed=seed),param_distributions=parameters,
                                     n_iter=n_iter,scoring="accuracy",
                                     n_jobs=-1,cv=8)

    import time as ttt
    now = time()
    log(ttt.ctime())
    rand_search.fit(X_train, y_train)
    report(rand_search.cv_results_, 10)
    log(ttt.ctime())
    log("Search took: " + str(time() - now))

python

best_fit(data_model.iloc[:, 1:], data_model.iloc[:, 0], n_iter=10)

For 10 runs and 12400s running time, the following parameters were discovered:

markdown

Model with rank: 1
Mean validation score: 0.550 (std: 0.034)
Parameters: {'n_estimators': 403, 'max_depth': 10, 'objective': 'binary:logistic', 'learning_rate': 0.15}

Model with rank: 2
Mean validation score: 0.546 (std: 0.044)
Parameters: {'learning_rate': 0.05, 'n_estimators': 201, 'objective': 'multi:softmax', 'max_depth': 10}

Model with rank: 2
Mean validation score: 0.546 (std: 0.035)
Parameters: {'n_estimators': 403, 'max_depth': 15, 'objective': 'binary:logistic', 'learning_rate': 0.1}

Model with rank: 4
Mean validation score: 0.545 (std: 0.033)
Parameters: {'n_estimators': 403, 'max_depth': 30, 'objective': 'binary:logistic', 'learning_rate': 0.15}

Model with rank: 5
Mean validation score: 0.543 (std: 0.033)
Parameters: {'max_depth': 30, 'n_estimators': 103, 'objective': 'multi:softmax', 'learning_rate': 0.3}

Model with rank: 6
Mean validation score: 0.542 (std: 0.041)
Parameters: {'n_estimators': 103, 'max_depth': 30, 'objective': 'binary:logistic', 'learning_rate': 0.1}

Model with rank: 7
Mean validation score: 0.542 (std: 0.046)
Parameters: {'n_estimators': 103, 'max_depth': 10, 'objective': 'binary:logistic', 'learning_rate': 0.05}

Model with rank: 8
Mean validation score: 0.541 (std: 0.033)
Parameters: {'learning_rate': 0.1, 'n_estimators': 103, 'objective': 'binary:logistic', 'max_depth': 10}

Model with rank: 9
Mean validation score: 0.509 (std: 0.068)
Parameters: {'n_estimators': 403, 'max_depth': 3, 'objective': 'multi:softmax', 'learning_rate': 0.1}

Model with rank: 10
Mean validation score: 0.504 (std: 0.068)
Parameters: {'max_depth': 3, 'n_estimators': 403, 'objective': 'multi:softmax', 'learning_rate': 0.15}

Test data classification

After finding best cross-validated parameter for the XGBoost, it’s time to load the test data and predict sentiment for them. Final classifier will be trained on the whole training set. End score will be revealed when the Angry Tweets competition will end (https://inclass.kaggle.com/c/angry-tweets).

The data will be exported to CSV file in format containing two columns: Id, Category. There are 4000 test samples with unknown distribution of the sentiment labels.

python

test_data = TwitterData()
test_data.initialize("data\\test.csv", is_testing_set=True)
test_data.build_features()
test_data.cleanup(TwitterCleanuper())
test_data.tokenize()
test_data.stem()
test_data.build_wordlist()
test_data.build_final_model(word2vec)

test_data.data_model.head(5)

	original_id	number_of_uppercase	number_of_exclamation	number_of_question	number_of_hashtags	number_of_mentions	...
0	628949369883000832	0	0	1	0	1	...
1	628976607420645377	1	1	0	0	1	...
2	629023169169518592	0	0	0	0	0	...
3	629179223232479232	0	0	0	0	0	...
4	629186282179153920	1	0	1	2	2	...

5 rows × 2398 columns

python

test_model = test_data.data_model

data_model = td.data_model

xgboost = XGBoostClassifier(seed=seed,n_estimators=403,max_depth=10,objective="binary:logistic",learning_rate=0.15)
xgboost.fit(data_model.iloc[:,1:],data_model.iloc[:,0])
predictions = xgboost.predict(test_model.iloc[:,1:])

python

test_model.head(5)

	original_id	number_of_uppercase	number_of_exclamation	number_of_question	number_of_hashtags	number_of_mentions	...
0	628949369883000832	0	0	1	0	1	...
1	628976607420645377	1	1	0	0	1	...
2	629023169169518592	0	0	0	0	0	...
3	629179223232479232	0	0	0	0	0	...
4	629186282179153920	1	0	1	2	2	...

5 rows × 2398 columns

python

results = pd.DataFrame([],columns=["Id","Category"])
results["Id"] = test_model["original_id"].astype("int64")
results["Category"] = predictions
results.to_csv("results_xgb.csv",index=False)

Feature importance

Let’s take a look at the final model feature importance.

Output list is really long, so the print line is commented in the code block

python

features = {}
for idx, fi in enumerate(xgboost.feature_importances_):
    features[test_model.columns[1+idx]] = fi

important = []
for f in sorted(features,key=features.get,reverse=True):
    important.append((f,features[f]))
    # print(f + " " + str(features[f]))

It’s not surprise that the word2vec and related good/bad/information similarity features were the most important, because the classification performance was greatly improved after switching to this representation.

What is interesting is that a lot of custom-crafted features (number_of_*) were also highly important, beating a lot of features which came from bag-of-words representation. Here’s the chart for the easiness of reading:

python

to_show = list(filter(lambda f: not f[0].startswith("word2vec") and not f[0].endswith("_similarity"),important))[:20]
to_show.reverse()
features_importance = [
    graph_objs.Bar(
        x=[f[1] for f in to_show],
        y=[f[0] for f in to_show],
        orientation="h"
)]
plotly.offline.iplot({"data":features_importance, "layout":graph_objs.Layout(title="Most important features in the final model",
      margin=graph_objs.Margin(
           l=200,
        pad=3
    ),)})

XGBoost considered custom features as important, so it also confirmed that some non-word features of a text can also be used to predict the sentiment. Most of them were even more important than the actual presence of some emotion-expressing words in the text.

Summary

Experiment showed that prediction of text sentiment is a non-trivial task for machine learning. A lot of preprocessing is required just to be able to run any algorithm and see - usually not great - results. Main problem for sentiment analysis is to craft the machine representation of the text. Simple bag-of-words was definitely not enough to obtain satisfying results, thus a lot of additional features were created basing on common sense (number of emoticons, exclamation marks etc.). Word2vec representation significantly raised the predictions quality. I think that a slight improvement in classification accuracy for the given training dataset could be developed, but since it contained highly skewed data (small number of negative cases), the difference will be probably in the order of a few percent. The thing that could possibly improve classification results will be to add a lot of additional examples (increase training dataset), because given 5971 examples obviously do not contain every combination of words usage, moreover - a lot of emotion-expressing words surely are missing.

Setup Apache Kafka in Docker on Windows

Marcin Zabłocki — Wed, 26 Apr 2017 22:00:00 +0000

This post describes how to quickly setup development environment for Apache Kafka in Docker on Windows. Apache Kafka is distributed streaming platform, which allows you to build reliable messaging channel between various systems. Its functionality supports publish/subscribe model, reliable message persistence, real-time stream processing and many more.

When I start learning any new technology, I want to get my hands drity as fast as possible, without spending too much time on environment setup. I wrote this post, to help you get started with Apache Kafka as fast as possible. In this post I cover full setup on Windows, but since it uses Docker, the process should be easily applicable for Linux/Mac machines. Besides Apache Kafka itself, the setup involves lauching Kafka-Manager UI (developed by Yahoo), so you will be able to manage your Kafka instance with ease. Let’s get started!

TL;DR

How to setup development environment for Apache Kafka + Kafka-Manager on Windows (using Docker).

Prerequisites

Docker installed (I use version 17.03.1-ce on Windows)
docker-compose installed (I use version 1.11.2)

Not required

knowledge of Docker
knowledge of Apache Kafka

Setup Apache Kafka with docker-compose

The setup will contain two containers:

Apache Kafka container (with Zookeeper built-in)
Kafka-Manger container

In order to join those two containers, we will use docker-compose. Create docker-compose.yml file and put the following content:

html

version: "2"

services:
  kafkaserver:
    image: "spotify/kafka:latest"
    container_name: kafka
    hostname: kafkaserver
    networks:
      - kafkanet
    ports:
      - 2181:2181
      - 9092:9092
    environment:
      ADVERTISED_HOST: kafkaserver
      ADVERTISED_PORT: 9092
  kafka_manager:
    image: "mzagar/kafka-manager-docker:1.3.3.4"
    container_name: kafkamanager
    networks:
      - kafkanet
    ports:
      - 9000:9000
    links:
      - kafkaserver
    environment:
      ZK_HOSTS: "kafkaserver:2181"

networks:
  kafkanet:
    driver: bridge

This docker-compose file decalres two mentioned containers. First one, kafkaserver runs Kafka image developed by Spotify team (it includes Zookeeper). It exposes itself under kafkaserver hostname in kafkanet network. Two environment variables need to be set:

ADVERTISED_HOST: kafkaserver
ADVERTISED_PORT: 9092

Those two describe to which hostname the Kafka server should respond from within the container.

Second container, kafka_manager runs newest (at the time of writing this post) kafka-manager tool. It needs to connect to the running Kafka server, that’s why there is the link entry in docker-compose:

html

links:
    - kafkaserver

Environment variable ZK_HOST also needs to be set in order to connect it to our Kafka server. We use hostname from kafkaserver container in order to achieve that.

Last thing in docker-compose file is network declaration, we use simple bridge networking there, so both containers can see one another.

Starting the environment

Once the docker-compose.yml file is ready, open your favorite terminal in the folder which contains it and run:

powershell

docker-compose up

After running this command, you can check status of the containers by invoking:

powershell

docker-compose ps

You should see output similar to this one:

Creating Kafka topic through Kafka-Manger UI

Now when the Kafka server is up and running, we can create a Kafka Topic. Topic in the Apache Kafka terminology is an isolated fragment to which multiple applications/systems can send messages and from which multiple consumers can receive (subscribe) data. One Kafka server can have any number of topics. I would say that Kafka Topic is equivalent of Table in SQL databases. It’s a separate category, isolated from the rest of the system.

Let’s launch Kafka-Manager, by opening

html

http://localhost:9000

in the browser:

Creating Kafka Cluster

As you can see, there are no Clusters yet (since it’s still an empty environment). Cluster is where Kafka stores all the data pushed to the topics. Let’s create one by using the Cluster menu:

For the quick setup, you can enter only:

Cluster Name - the name of your choice
Cluster Zookeeper Hosts - hostname of zookeeper (which is kafkaserver as configured in docker-compose.yml file)
Kafka Version 0.10.1.0 - which is the latest version at the time of writing this post

After entering those, click Save. You should be redirected to the homepage and see the cluster there:

Creating Kafka Topic

Now, when the cluster is present, navigate to it’s properties and open Create Topic from the menu:

Entering only Topic name is enough. Just hit Create from there. Done! Topic is created!

Testing Apache Kafka on Windows

It is done, environment is up and running. Let’s test it. The quickest way to do this is to use Apache-provided scripts and just launching two of them - one for the producer and one for the subscriber. In order to get those scripts, download the folowing file (it’s the actual Apache Kafka distribution, but we only need scripts from it):

html

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz

and extract it. Then follow the path to the following folder:

powershell

kafka_2.11-0.10.2.0.tgz\kafka_2.11-0.10.2.0.tar\kafka_2.11-0.10.2.0\bin\windows\

This folder contains bunch of script files:

powershell

Name
----
connect-distributed.bat
connect-standalone.bat
kafka-acls.bat
kafka-configs.bat
kafka-console-consumer.bat
kafka-console-producer.bat
(...)
zookeeper-server-start.bat
zookeeper-server-stop.bat
zookeeper-shell.bat

We are interested in kafka-console-consumer.bat and kafka-console-producer.bat, but before launching them, we need one more thing. In order to connect to Kafka in Docker from the Windows host, we need to map Kafka container hostname to the localhost, by editing hosts file:

powershell

C:\Windows\System32\drivers\etc\hosts

Append this line:
127.0.0.1 kafkaserver

Ok, let’s launch Kafka Consumer first:

powershell

K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic kafka_test
_topic

After launching this script, PowerShell window will hang, that’s a good thing. Open second terminal and launch Kafka Producer:

powershell

PS K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-producer.bat --broker-list localhost:9092 --topic kafka_test_topi
c

Type any message and hit ENTER, you should see the messages flowing, like this:

Stopping Apache Kafka docker

When the work is done, you can easily turn off running containers by invoking:

powershell

docker-compose down

in the folder when you have your docker-compose.yml file:

Summary

Now you have Apache Kafka with Kafka-Manger development environment up and running on your Windows machine. I hope that this short post helped you with the setup. Please share or leave a comment if you liked it!

Additional links & resources

full docker-compose.yml file used in this post: docker-compose.yml
Spotify Apache Kafka docker image: https://github.com/spotify/docker-kafka
Kafka-Manager docker image: https://hub.docker.com/r/mzagar/kafka-manager-docker/

Setup ASP.NET Core 1.0 debugging on Ubuntu 16.04

Marcin Zabłocki — Tue, 24 May 2016 22:00:00 +0000

There has been a lot of excitement about running ASP.NET Core 1.0 applications on Linux. Since RC2, Microsoft provided official support for running .NET Core on Ubuntu, however up to the date of writing this post, .NET Core is officially working only on Ubuntu 14.04. What about newest version of this OS? You can easily get it work (with debugging!). Here’s the tutorial, how to run and debug ASP.NET Core 1.0 RC2 on Ubuntu 16.04.

Prerequsites

Ubuntu 16.04 installed (can be a VM)
Visual Studio Code with latest updates installed
good will

Installing .NET Core on Ubuntu 16.04 (dotnet)

Run the following commands to add the following repository package sources to your system (taken from the official Microsoft’s http://dot.net site, from Linux/Ubuntu section):

powershell

sudo sh -c 'echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/dotnet/ trusty main" > /etc/apt/sources.list.d/dotnetdev.list'

sudo apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893

sudo apt-get update

Now, the original steps requires to run:

powershell

sudo apt-get install dotnet-dev-1.0.0-preview1-002702

But… there will be an error:

powershell

The following packages have unmet dependencies:
 dotnet-dev-1.0.0-preview1-002702 : Depends: dotnet-sharedframework-microsoft.netcore.app-1.0.0-rc2-3002702 but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

So, let’s fix it! Trying to install the dependant package dotnet-sharedframework-microsoft.netcore.app will get you next error, saying that this package depends on libicu52 but it is not installable. Really? Head to:

powershell

http://packages.ubuntu.com/pl/trusty/amd64/libicu52/download

and download .deb package. Now, go to the download location and install the package by running:

powershell

sudo dpkg -i libicu52_52.1-3ubuntu0.4_amd64.deb

Now, you will be able to run the following commands:

powershell

sudo apt-get install dotnet-sharedframework-microsoft.netcore.app-1.0.0-rc2-3002702
sudo apt-get install dotnet-dev-1.0.0-preview1-002702

A few moments and…

powershell

dotnet --version

…yay! It’s installed.

Visual Studio Code ASP.NET Core Debugging on Ubuntu 16.04

Now, when you have dotnet installed, it’s time to setup the debugger. If already have a project to debug, that’s great. If not - download official samples from ASP.NET github repository (https://github.com/aspnet/cli-samples - remember to run dotnet restore in the project directory afterwards). After setting this up, just open project directory in VS Code.

First of all, you need Visual Studio Code C# extension. To install it, just press Ctrl+E inside VS Code and type:

powershell

ext install C#

After sucessful installation, editor will prompt you to restart. And now… another error! It says, that Ubuntu 16.04 is not supported by .NET Core Debugger. Or is it? ;]

To HACK this issue, go back to the terminal, and head to

powershell

/etc/

In this directory, there is a lot of files… And here comes the greatness of “everything-is-a-file” Linux philosophy: the /etc/ directory contains the following files:

powershell

os-release
lsb-release

Which as you might have guessed - store the information about Ubuntu version. Let’s change all of the 16.04 to 14.04 and see what happens!

powershell

sudo nano os-release

edit the file, press Ctrl+O [ENTER] to save, Ctrl+X to exit
repeat the same procedure for the next file:

sudo nano lsb-release

I provide you my files if you are uncertain how they should look:

/etc/lsb-release

html

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS"

/etc/os-release

html

NAME="Ubuntu"
VERSION="14.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial

Go back to Visual Studio Code. Oh my goodness! .NET Core Debugger have installed! VS Code will also prompt you to add debug configuration file, click yes to generate it.

Debugging ASP.NET Core 1.0 on Ubuntu 16.04

The final step. Open debug view in Code and make sure that you have .NET Core Launch (console) selected as debug target.

Start debugging by pressing F5. Go back to your code, and mark breakpoint somewhere. The ultimate test - open web browser and hit localhost:5000 (or any other port if you have changed it)… drumroll please… Isn’t it awesome?

I hope this post helped you to fully setup ASP.NET Core debugging on Ubuntu 16.04. Happy C# coding… on Linux!

Sitecore Dynamic Field Editor Custom Experience Button

Marcin Zabłocki — Tue, 19 Apr 2016 22:00:00 +0000

Recently, I’ve worked on the Sitecore project which required author-friendly editing of item fields. Although Sitecore CMS has Field Editor feature, but I wanted something more. In this post I will show you how to rapidly implement custom experience button which will allow to dynamically choose fields depending on any business logic required.

I’ve taken the rapid development approach (which is often required when the deadline is approaching :-) and I took advantage of existing Sitecore’s Field Editor to extend it a little bit. Of course, the built-in Field Editor have the ability to display various fields, but it’s based only on the static list of fields stored in the Field Editor Button Item. It does not meet my requirements - truly dynamic field list. So… I’ve sat with the laptop and did the work.

Here is a list of features of my Dynamic Field Editor:

execute any business logic to provide dynamic field list
dynamically show or hide sections
high reusability - can be used with any rendering/sublayout without adding any new items/commands

If you are TL;DR person, take a look at this screencap and see if this functionality suits your needs:

How this will work?

My idea to implement this was to create new custom experience button and then replacing the webedit:fieldeditor command by my own. To make this highly reusable and extensible, the replacement command will invoke custom pipeline. The pipeline will have processors for every rendering/sublayout with the business logic. In this implementation, an assumption is made that the pipeline processors will modify Field Editor options only if the command was invoked from specific rendering/sublayout.

Prerequsites

Sitecore 8.X (I did this on 8.1-update 2, but it should work on any other version as well)
Visual Studio project set up for Sitecore (blank web app with references Sitecore DLLs)

Creating replacement command

First, we will implement custom web edit command which will invoke the pipeline. In the project, create a class which inherits from the Sitecore class:

csharp

Sitecore.Shell.Applications.WebEdit.Commands.FieldEditor

I always want to be as flexible as possible inside of my code and do not end up with “aww… I don’t know XYZ property at this point”, so my implementation takes as much as possible from the command context and passes it to custom pipeline. Here’s what it looks like:

csharp

public class DynamicFieldEditor : FieldEditor
{
    protected CommandContext CommandContext;

    public override void Execute(CommandContext context)
    {
        CommandContext = context;
        base.Execute(context);
    }

    protected override PageEditFieldEditorOptions GetOptions(ClientPipelineArgs args, NameValueCollection form)
    {
        var options = base.GetOptions(args, form);
        var pipelineArgs = new GetDynamicFieldEditorPipelineArgs(args, form, CommandContext, options);
        CorePipeline.Run("getDynamicFieldEditor", pipelineArgs, false);
        return options;
    }
}

Now, we need to edit configuration file to replace original command:

html

 <commands>
    <command name="webedit:fieldeditor" type="Zablo.Net.Sitecore.Commands.DynamicFieldEditor, Zablo.Net.Sitecore" patch:instead="*[@name='webedit:fieldeditor']"></command>
</commands>

Custom pipeline - abstraction

The pipelines need two things: Pipeline Args and Pipeline Interface. As you could see above, my pipeline will be called getDynamicFieldEditor and it operates on the custom arguments class. Arguments class have as much context-related properties as possible, so the processors will be able to modify virtually anything.

csharp

public class GetDynamicFieldEditorPipelineArgs : PipelineArgs
{
    public GetDynamicFieldEditorPipelineArgs(ClientPipelineArgs args, NameValueCollection form, CommandContext commandContext, PageEditFieldEditorOptions options)
    {
        ClientPipelineArgs = args;
        Form = form;
        CommandContext = commandContext;
        FieldEditorOptions = options;
    }

    public ClientPipelineArgs ClientPipelineArgs { protected set; get; }

    public NameValueCollection Form { protected set; get; }

    public CommandContext CommandContext { protected set; get; }

    public PageEditFieldEditorOptions FieldEditorOptions { protected set; get; }
}

The interface is pretty simple too:

csharp

public interface IGetDynamicFieldEditorProcessor
{
    void Process(GetDynamicFieldEditorPipelineArgs args);
}

Custom pipeline - base implementation

I hope that you didn’t stopped at this point and started implementing your own processor. There is more - I’ve implemented base processor, which extracts even more data - like the rendering/sublayout datasource item - to make development of this pipeline’s processors even faster. Here you go:

csharp

public abstract class BaseDynamicFieldEditorProcessor : IGetDynamicFieldEditorProcessor
{
    protected abstract ID RenderingId { get; }
    public virtual void Process(GetDynamicFieldEditorPipelineArgs args)
    {
        var parameters = args.ClientPipelineArgs.Parameters;
        var renderingId = parameters.Get("renderingId");
        var options = args.FieldEditorOptions;

        if (string.IsNullOrEmpty(renderingId) || ID.Parse(renderingId) != RenderingId)
        {
            return;
        }
        var datasourceUri = parameters["uri"];
        Item datasource = null;
        if (datasourceUri != null)
        {
            datasource = Database.GetItem(ItemUri.Parse(datasourceUri));
        }
        ProvideDynamicFields(args, options, parameters, datasource);
    }

    /// <summary>
    /// Provide custom logic for displaying fields here
    /// </summary>
    /// <param name="args">
    /// <param name="options">
    /// <param name="parameters">
    /// <param name="datasource">
    protected abstract void ProvideDynamicFields(GetDynamicFieldEditorPipelineArgs args,
        PageEditFieldEditorOptions options, NameValueCollection parameters, Item datasource);
}

The first condition in Process method is really important. It matches my assumption, that specific processor can only proceed execution only if it is invoked from it’s related rendering/sublayout.

Custom pipeline - processor for rendering/sublayout

Like I said before, you can apply my Dynamic Field Editor to any rendering/sublayout, so I will show you how to implement processor for Sitecore’s built-in Sample Rendering.

First, create class which inherits from BaseDynamicFieldEditorProcessor, then open the configuration file from your solution, and add first processor to the getDynamicFieldEditor pipeline:

html

<pipelines>
    <getdynamicfieldeditor>
    <processor type="Zablo.Net.Sitecore.Pipelines.GetDynamicFieldEditor.SampleRenderingFieldProcessor, Zablo.Net.Sitecore"></processor>
    </getdynamicfieldeditor>
</pipelines>

There are two things to implement:

RenderingId property getter
ProvideDynamicFields method

The property can be hard-coded (like mine here), but there is no obstacle in calling the configuration, to get your renderingId from config XML or any other place. The method focuses only on adding fields to the Field Editor. Here is a full implementation:

csharp

public class SampleRenderingFieldProcessor : BaseDynamicFieldEditorProcessor
{
    protected override ID RenderingId { get { return ID.Parse("{493B3A83-0FA7-4484-8FC9-4680991CF743}"); } }

    protected override void ProvideDynamicFields(GetDynamicFieldEditorPipelineArgs args, PageEditFieldEditorOptions options,
        NameValueCollection parameters, Item datasource)
    {
        options.Fields.Clear();
        if (datasource.TemplateID == ID.Parse("{76036F5E-CBCE-46D1-AF0A-4143F9B557AA}"))
        {
            options.ShowSections = false;
            options.PreserveSections = false;
            options.Fields.Add(new FieldDescriptor(datasource,"Title"));
        }
        else
        {
            options.ShowSections = true;
            options.PreserveSections = true;
            options.Fields.Add(new FieldDescriptor(datasource, "SomeOtherField"));
        }
    }
}

There’s nothing fancy it the logic above - just template checking. I didn’t wanted to obfuscate the code, just wanted to show the idea of implementation.

Adding Custom Experience Button

Last thing to do is to add the Custom Experience Button. Head to the CORE database and duplicate the item from path:

html

/sitecore/content/Applications/WebEdit/Edit Frame Buttons/Default/Edit

and move it to:

html

/sitecore/content/Applications/WebEdit/Custom Experience Buttons

I’ve also renamed it to Dynamic Field Editor and changed the icon - which was obviously the most important task ;) Make sure that Field List field of duplicated item is not empty, because Sitecore will throw exception otherwise. Finally, switch back to MASTER database, find your rendering and add the Custom Experience Button to it (the field is located under Editor Options section):

Summary

That’s it! I hope that Dynamic Field Editor and it’s extensibility will give you an opportunity to improve Sitecore author experience without lot of custom development. I will appreciate your feedback, please comment and share if you like it!

Full source code can be found on my github:

https://github.com/marrrcin/sitecore-dynamic-field-editor

Modular ASP.NET Core 1.0 project structure

Marcin Zabłocki — Mon, 21 Mar 2016 23:00:00 +0000

ASP.NET Core 1.0 (formely ASP.NET 5) introduced some new framework concepts as well as cut loose the project structure, giving the developers ability to fully customize it. In example, we are no more bound to the Areas folder structure, which in my opinion unnecessarily duplicated Controllers/Models/Views folder structure in each of them.

Moreover, new version of the framework added a feature called View Components, which finally resolved issue of having additional controllers returning only partial views. From now on, we can make true components in ASP.NET without any workarounds.

I’ve already written a few projects in ASP.NET Core 1.0 and here is how I worked my way into creating elastic project structure.

Well structured areas

First of all - the areas. They are in ASP.NET MVC for quite a while and they are really useful. Personally, I’ve used them every time when I had to write application with both admin panel and client facing pages. The bigger part usually landed inside main project structure and the smaller one inside the Areas folder. And… I hated that. I wanted Controllers inside the Controllers folder, Views inside Views folder and Models inside Models folder. Finally - I can do that really easily.

Let’s say we have three areas: Main (client facing), Administrator and Content Authors. The output of the following configuration will look like this.

html

<projectroot>
└───Controllers
    └───Admin
    └───Authors
    └───HomeController.cs
    └───(...) other client controllers
└───Views
    └───Admin
    └───Authors
    └───Shared
    └───Home
        └───Index.cshtml
        └───(...) other client views
└───Models
    └───Admin
    └───Authors
    └───(...)HomePageViewModel.cs

As you can see, Admin and Author controllers are placed inside Controllers folder - it is where you would start looking for any controller in somebody else’s project, don’t you? The same rule applies for the Views and Models (actually - View Models) folders.

To make this structure work, the only thing we need to do is to extend Razor View Engine, so it will know how to find Views for particular Area. This can be done by implementing IViewLocationExpander interface:

csharp

public class ViewLocationExpander : IViewLocationExpander
{
    public void PopulateValues(ViewLocationExpanderContext context)
    {
        // nothing here
    }

    public IEnumerable<string> ExpandViewLocations(ViewLocationExpanderContext context, IEnumerable<string> viewLocations)
    {
        var area =
            context.ActionContext.ActionDescriptor.RouteConstraints.FirstOrDefault(rc => rc.RouteKey == "area");
        var additionalLocations = new LinkedList<string>();
        if (area != null)
        {
            additionalLocations.AddLast($"/Views/{area.RouteValue}" + "{1}/{0}.cshtml");
        }
        return viewLocations.Concat(additionalLocations);
    }
}

Now we need to attach this class to Razor engine. Head to Startup.cs and find the line in which you add MVC to application services:

csharp

services.AddMvc();

and extend this line in order to add Razor View Engine Options:

csharp

services.AddMvc().AddRazorOptions(razorOptions =>
{
    razorOptions.ViewLocationExpanders.Add(new ViewLocationExpander());
});

From now on, for every controller marked with [Area] atrribute, the framework will search for it’s views inside proper folder located under projects Views folder. Of course, this solution requires to have exactly the same name in the folder structure as in the attribute parameter. If you want to have different area name and different folder names, just add some if/switch inside of the ExpandViewLocations method and you will be good to go.

Reusable & isolated components

Some time ago, somebody asked on Stackoverflow, where the JavaScript files for the View Components should be placed in ASP.NET Core 1.0 projects. I’ve responed for that question, and here is my point of view (link to the answer bellow the post).

Isolation of components should not only consider JavaScript files, but also backing code (ViewComponent class) and the component’s view razor file. In my projects, I tend to make this kind of structure for View Components:

markdown

<projectroot>
└───Components
    └───AwesomeComponent
        └───Awesome.cshtml
        └───AwesomeComponent.cs
        └───AwesomeComponentModel.cs
    └───CommentsComponent
        └───Comments.cshtml
        └───CommentsComponent.cs
        └───CommentsComponentModel.cs

I treat every component as totally isolated entity. It is concise and it only contains necessary files. I can easily cut the component from my project and put it into another. For example, the Comments Component (actually used on this blog to show Discuss) just renders the required HTML with some dynamically rendered parameters passed to it’s view via CommentsComponentModel (which is the View Model).

View, Backend code and Model are all in one place, so when you move around the component’s folder, you are sure that you move whole component. Moreover, when you are modyfying them, you have quick access to all of their parts.

It will be convinient to put JavaScript and/or CSS files which are highly coupled with a component also in such structure. You can do this by simply creating the file under the component’s folder, and then writing a GULP TASK that will copy JS/CSS files to wwwroot. From that point, you will be able to link that JavaScript code on component’s .cshtml using standard syntax:

html

<script src="~/Components/awesomecomponent.js"></script>

After such setup, in order to render View Component with it’s parameters, use the following syntax:

csharp

@await Component.InvokeAsync("AwesomeComponent",new AwesomeComponentModel())

And remember to put [ViewComponent(Name = “Awesome”)] annotation on the top of component’s backing class if you want to avoid adding full (Name)ViewComponent convention and just have (Name)Component classes.

Additional links

My Stackoverflow anwser for “Where should my Javascript go for MVC 6 view components?” http://stackoverflow.com/questions/34932699/where-should-my-javascript-go-for-mvc-6-view-components/34933221#34933221

ASP.NET Core Redis HTML Cache

Marcin Zabłocki — Tue, 08 Mar 2016 22:51:42 +0000

Previous versions of ASP.NET MVC framework had out-of-the-box OutputCache attribute, which could be used to improve MVC aplications performance by - as the name describes - caching the output of actions to which the attribute was added.

In ASP.NET Core 1.0 (formerly ASP.NET 5) the attribute no longer exists. The ASP.NET team provided ResponseCache attribute, however it's purpose is only to set HTTP response headers, so the proxy servers behind which the web application could be running, could cache the response. What about local caching? So far, I didn't found out-of-the-box solution in the edge version of the framework (which is as of writing this post in RC1 state).

So... the need for that functionality became the code in my Visual Studio. I will show you how I've implemented full HTML cache for ASP.NET Core 1.0 using REDIS cache (and also in-proc memory), and I'll be glad to read comments about my solution and also discuss about it. Here it goes!

Implementing ASP.NET Core HTML cache

Prerequisites:

ASP.NET 5 (ASP.NET Core 1.0) RC1 installed
Visual Studio 2015
access to any Redis server*
ASP.NET dnx451 targeted project (there is no .NET Core support for Redis Cache yet)

*Redis is obviously optional when you use memory cache

Preparation

My cache solution will be a tandem of ASP.NET Middleware and C# Attribute usage. Let's start with creating new Web Application project. You will also need some additional packages, so open project.json and add those:

"Microsoft.Extensions.Caching.Memory": "1.0.0-rc1-final",
"Microsoft.Extensions.Caching.Redis": "1.0.0-rc1-final"

If Visual Studio fails to restore packages, make sure that you have proper package sources. Those are mine: (Nuget.config)

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
        <add key="api.nuget.org" value="https://api.nuget.org/v3/index.json" />
        <add key="AspNetVNext" value="https://www.myget.org/F/aspnetvnext/api/v2/"
        />
        <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
        <add key="Roslyn" value="https://www.myget.org/F/roslyn-nightly/" />
        <add key="DotNet" value="https://www.myget.org/F/dotnet-corefx/" />
    </packageSources>
</configuration>

How it will work

Before writing the code, here is a description how my solution will work:

Adding custom created Cache attribute adds action filter to marked action
Action filter handles those events:
1. OnActionExecuting - to prevent action executing if cached response exists, and to write cached response into current request result
2. OnResultExecuting - to finish request handling, when cache is hit
3. OnResultExecuted - when there was a cache miss, to save response into cache
Custom middleware to intercept response stream and provide own MemoryStream to which custom code will be able to write and read.w

Implementation - abstract layer

After restoring packages and setting target framework - add new folder to your project, call it Cache - all of cache related code will land there.

In order to have some abstraction and avoid hard dependencies, I've created ICacheService interface, which exposes general purpose cache methods:

public interface ICacheService
{
    void Store(string key, object content);

    void Store(string key, object content, int duration);

    T Get(string key) where T : class;
}

Implementation of this service will be described later in this post.
Next - let's add CacheAttribute class, which will implement IActionFilter and inherit from ResultFilterAttribute.

public class CacheAttribute : ResultFilterAttribute, IActionFilter
{
//...
}

Custom middleware for accessing response stream

Before writing the implementation of this attribute, we will need access to the response stream (which is somehow locked to write/read by default). In order to obtain this functionality, add the middleware class CacheMiddleware.cs. Every middleware class requires non-interfaced method with the signature as follows:
public async Task Invoke(HttpContext httpContext)

By desing, every middleware must invoke further registered middlewares, there is no exception for this one. Easiest way to do this is to add RequestDelegate dependency in constructor, so build-in ASP.NET DI container will inject this object into our middleware.

Purpose of CacheMiddleware is to intercept request, and provide custom stream to which further middlewares could write (and read). When the middleware invocation process will return to CacheMiddleware, this stream's content will be copied to original response stream. Full implementation of this class:

public class CacheMiddleware
{
	protected RequestDelegate NextMiddleware;

	public CacheMiddleware(RequestDelegate nextMiddleware)
	{
		NextMiddleware = nextMiddleware;
	}

	public async Task Invoke(HttpContext httpContext)
	{
		using (var responseStream = new MemoryStream())
		{
			var fullResponse = httpContext.Response.Body;
			httpContext.Response.Body = responseStream;
			await NextMiddleware.Invoke(httpContext);
			responseStream.Seek(0, SeekOrigin.Begin);
			await responseStream.CopyToAsync(fullResponse);
		}
	}
}

To make use of this class, we need to add it to the request pipeline. In Startup.cs file, in Configure method, just before app.UseMvc(), add:
app.UseMiddleware<CacheMiddleware>();

Cache attribute

We are now set to write CacheAttribute implementation. First, add those properties to the CacheAttribute.cs class:

protected ICacheService CacheService { set; get; }

public int Duration { set; get; }

Duration will be handy if there is any need for adjustment of cache time. Usage of CacheService property will be described shortly.

Let's move along with caching output in OnResultExecuted method:

public override void OnResultExecuted(ResultExecutedContext context)
{
	GetServices(context);

	var cacheKey = context.HttpContext.Request.GetEncodedUrl().ToMd5();
	var httpResponse = context.HttpContext.Response;
	var responseStream = httpResponse.Body;

	responseStream.Seek(0, SeekOrigin.Begin);

	using (var streamReader = new StreamReader(responseStream, Encoding.UTF8, true, 512, true))
	{
		var toCache = streamReader.ReadToEnd();
		var contentType = httpResponse.ContentType;
		var statusCode = httpResponse.StatusCode.ToString();
		Task.Factory.StartNew(() =>
		{
			CacheService.Store(cacheKey + "_contentType", contentType, Duration);
			CacheService.Store(cacheKey + "_statusCode", statusCode, Duration);
			CacheService.Store(cacheKey, toCache, Duration);
		});

	}
	base.OnResultExecuted(context);
}

What it does? First - it uses Service Locator Pattern (which you could obviously replace with proper DI-way, it was used here for the quick implementation purposes) to get access to CacheService. Then, it takes the response stream (which was provided by our custom middleware) and reads the whole of it (with leaving the stream open, thus the monstrous StreamReader constructor usage). After all of this, method stores response, response type and status code in the Cache - this process is run in the background, so the request could process further without any blocking operations. At last, the OnResultExecuted method invokes base method, so any remaining filters can execute.

Now, when we have response cached, we need a way to intercept request and write it to the response. There goes the implementation of OnActionExecuting method:

public void OnActionExecuting(ActionExecutingContext context)
{
	GetServices(context);
	var requestUrl = context.HttpContext.Request.GetEncodedUrl();
	var cacheKey = requestUrl.ToMd5();
	var cachedResult = CacheService.Get<string>(cacheKey);
	var contentType = CacheService.Get<string>(cacheKey + "_contentType");
	var statusCode = CacheService.Get<string>(cacheKey + "_statusCode");
	if (!string.IsNullOrEmpty(cachedResult) && !string.IsNullOrEmpty(contentType) &&
		!string.IsNullOrEmpty(statusCode))
	{
		//cache hit
		var httpResponse = context.HttpContext.Response;
		httpResponse.ContentType = contentType;
		httpResponse.StatusCode = Convert.ToInt32(statusCode);

		var responseStream = httpResponse.Body;
		responseStream.Seek(0, SeekOrigin.Begin);
		if (responseStream.Length <= cachedResult.Length)
		{
			responseStream.SetLength((long)cachedResult.Length << 1);
		}
		using (var writer = new StreamWriter(responseStream, Encoding.UTF8, 4096, true))
		{
			writer.Write(cachedResult);
			writer.Flush();
			responseStream.Flush();
			context.Result = new ContentResult { Content = cachedResult };
		}
	}
	else
	{
		//cache miss
	}
}

As you can see, again - I take CacheService. This time, the code searches for the same cache keys used to store response. If all keys are found (cache hit). From this point, we set response Content Type, Status Code and finally - we write cached HTML into response stream. As you can see, there is a part of code which extends stream's length. I've encountered an issue in which I could not write to the stream (because of the length issues), so I've just extended the length of it. The line which sets context.Result is very important, as the implementation of next method - OnResultExecuting - relays on it:

public override void OnResultExecuting(ResultExecutingContext context)
{
	if (context.Result is ContentResult)
	{
		context.Cancel = true;
	}
}

This code cancels the request pipeline, if there was a Cache Hit. Of course, you can create more sophisticated condition here - I'm sure that it will not work in some cases, but this post is about general idea of HTML cache, not perfect-error-resistant-uber-implemntation.

OnResultExecuted method will be left blank.

Implementing Cache Service

We are almost done. The only missing part is the actual CacheService. As promised in the title of this post - Redis cache will be used. I will also provide my implementation of MemoryCache, so you can test HTML cache without Redis server.

public class RedisCacheService : ICacheService
{
	protected IDistributedCache Cache;
	private static int DefaultCacheDuration => 60;

	public RedisCacheService(IDistributedCache cache)
	{
		Cache = cache;
	}

	public void Store(string key, object content)
	{
		Store(key, content, DefaultCacheDuration);
	}

	public void Store(string key, object content, int duration)
	{
		string toStore;
		if (content is string)
		{
			toStore = (string)content;
		}
		else
		{
			toStore = JsonConvert.SerializeObject(content);
		}

		duration = duration <= 0 ? DefaultCacheDuration : duration;
		Cache.Set(key, Encoding.UTF8.GetBytes(toStore), new DistributedCacheEntryOptions()
		{
			AbsoluteExpiration = DateTime.Now + TimeSpan.FromSeconds(duration)
		});
	}

	public T Get<T>(string key) where T : class
	{
		var fromCache = Cache.Get(key);
		if (fromCache == null)
		{
			return null;
		}

		var str = Encoding.UTF8.GetString(fromCache);
		if (typeof(T) == typeof(string))
		{
			return str as T;
		}

		return JsonConvert.DeserializeObject<T>(str);
	}
}

Implementation depends on IDistributedCache - interface provided by the ASP.NET Core 1.0 framework itself. To use it, we need to configure it. Head to the Startup.cs file again, but this time, go to ConfigureServices method and add this code:

services.AddCaching();
string redisConnection = Configuration["Redis:ConnectionString"],
	   redisInstance = Configuration["Redis:InstanceName"];

services.AddSingleton<IDistributedCache>(factory =>
{
	var cache = new RedisCache(new RedisCacheOptions
	{
		Configuration = redisConnection,
		InstanceName = redisInstance
	});

	return cache;
});

You can see that this code uses some configuration data. By default, ASP.NET Core 1.0 configuration is stored in appsettings.json file. Open it and configure your Redis instance connection strings:

"Redis": {
    "ConnectionString": "redis-server:7331,password=correcthorsebatterystaple",
    "InstanceName": "redis"
}

Last thing to do is to add implemented CacheAttribute and RedisCacheService to the application. Just add those lines bellow the previous Redis-related ones in Startup.cs:

services.AddSingleton<ICacheService, RedisCacheService>();
services.AddTransient<CacheAttribute>();

Testing the solution

OK, let's test it! I've replaced original Home/Index.cshtml with this code:

@model dynamic
@DateTime.Now
<script type="text/javascript">
    document.write("<p>" + new Date().toLocaleString() + "</p>");
</script>

and then I've added CacheAttribute to HomeController Index method:

[Cache(Duration = 30)]
public IActionResult Index()
{
	return View();
}

Now - let's start the application using dnx web (or dnx-watch web) command and open http://localhost:5000 (default URL). First request processes along the whole pipeline, but the second one responds from cache, as you can see in the screenshots bellow.

Full request

Cached request

Redis server view

Using memory to cache HTML instead of Redis

As my implementation is interface-only dependant, the RedisCacheService could be easily replaced by In-memory Cache. Here is the implementation:

public class MemoryCacheService : ICacheService
{
	protected IMemoryCache Cache;

	public MemoryCacheService(IMemoryCache cache)
	{
		Cache = cache;
	}

	public void Store(string key, object content)
	{
		Store(key, content, DefaultCacheDuration);
	}

	public void Store(string key, object content, int duration)
	{
		object cached;
		if (Cache.TryGetValue(key, out cached))
		{
			Cache.Remove(key);
		}

		Cache.Set(key, content,
			new MemoryCacheEntryOptions
			{
				AbsoluteExpiration = DateTime.Now + TimeSpan.FromSeconds(duration),
				Priority = CacheItemPriority.Low
			});
	}


	private static int DefaultCacheDuration => 60;

	public T Get<T>(string key) where T : class
	{
		object result;
		if (Cache.TryGetValue(key, out result))
		{
			return result as T;
		}
		return null;
	}
}

Wrap up

And that's it. I hope you like my post, please share or/and leave a comment! Of course, the whole source code can be found on github: https://github.com/marrrcin/aspnet-core-redis-html-cache
Please comment and share if you like it!

Marcin Zabłocki — Mon, 01 Jan 0001 00:00:00 +0000

Marcin Zabłocki blog

How to connect to Home Assistant from anywhere using NordVPN meshnet

What is NordVPN Meshnet?

How it works in practice

Quick solution summary

Meshnet configuration on iOS

Prerequisites

Diagnostic steps: How to debug “Home Assistant not working on meshnet”

Check if Home Assistant is actually listening

Verify the local network path works first

Test the meshnet connection itself

Understand why SSH works but Home Assistant doesn’t

The complete solution: step-by-step

Step 1: enable UFW

Step 2: allow meshnet traffic on port 8123

Step 3: reload UFW

Step 4: grant meshnet local network access

Step 5: test the connection

That’s all

Kubernetes: podAffinity with NotIn vs podAntiAffinity with In

The Question

The Answer

Deep walk-through

DON’T BELIEVE ME, CHECK IT OUT BY YOURSELF! 👀

Hands-On Example: Experimenting with Pod Affinity and Anti-Affinity

Step 1: Install k3d

Step 2: Create a k3d Cluster

Step 3: Set Up kubectl

Step 4: Experiment with Pod Affinity and Anti-Affinity

Optional: Scale Up and Add Nodes

Clean Up

Key Takeaways

Don’t Let This Happen to You

Twitter Sentiment Analysis on BigQuery using ONNX + Huggingface Transformers

TL;DR

What is not in the scope of this blogpost

What you will find below

Prerequisites

Dataset

Preparing the model

How to fix it? - take 1

How to fix it? - take 2

Creating BigQuery ONNX model

Huggingface tokenizers in BigQuery

Huggingface tokenizer as BigQuery Remote Function

DistilBERT inference on BigQuery

Scalling the inference

ONNX models vs Remote Functions

Summary

Additional links & resources

Deploy MLflow models on BigQuery

TL;DR

⚠️ What is not in the scope of this blogpost

What you will find below

Prerequisites

Training a simple model with TPOT’s AutoML

Creating BigQuery Remote Function with Fast API

BigQuery Remote Function contract

Remote Function endpoint

Testing out the endpoint locally

Deploying Remote Function to Cloud Run

Building a Docker container

Create Cloud Run service

Connecting from BigQuery to Cloud Run

(option 1) From UI

(option 2) From CLI

Running the inference from BigQuery

💡Pro-tips

MLflow model saving

Multiple Remote Functions in a single endpoint

Summary

Additional links & resources

Robust Subspace Recovery Layer implementation in PyTorch

TL;DR

Idea behind Robust Space Recovery Layer

What is the RSR Layer?

How to implement the RSR Layer in PyTorch?

RSR Auto Encoder in PyTorch

Training the autoencoder with RSR Layer in PyTorch (& PyTorch Lightning)

Corrupted MNIST dataset (with outliers)

Step 3: Set Up `kubectl`

2. Stop using `.to(device)` that much: `zeros_like` / `ones_like` etc.

3. Register Buffer (a.k.a `nn.Module.register_buffer`)

4. Built-in `Identity()`

5. Pairwise distances: `torch.cdist`

6. Cosine similarity: `F.cosine_similarity`

7. Normalizing vectors: `F.normalize`

8. Linear layer + chunking trick (`torch.chunk`)

9. Masked select (`torch.masked_select`)

10. Conditional tensors with `torch.where`

11. Filling tensors with values at given positions (`Tensor.scatter`)

12. Image interpolation within network (`F.interpolate`)

13. Making grid of images (`torchvision.utils.make_grid`)