machine-learning on Marcin Zabłocki blog

Twitter Sentiment Analysis on BigQuery using ONNX + Huggingface Transformers

Marcin Zabłocki — Sun, 14 May 2023 01:00:00 +0000

This is my second blog post about running machine learning / deep learning models in BigQuery. This time, I will use the latest (still in preview) capabilities of BigQuery ML that allow to run ONNX models within the BigQuery itself.

In this blogpost dig into this new feature, to explore it’s capabilities and limitations (there are few!). As an example, I will deploy a DistilBERT-based classifier to analyse the sentiment of tweets stored in a BigQuery table.

If you’re interested in deploying custom models to BigQuery (that cannot be converted to ONNX), check out my previous blogpost: Deploy MLflow Models On BigQuery.

TL;DR

You will learn how to deploy ONNX models to BigQuery ML & understand the limitations.

What is not in the scope of this blogpost

For brevity, I’m not training any models here - I’m reusing a publicly available model.

What you will find below

Prerequisites

You will need:

Python (I’m using 3.10)
access to BigQuery
access to Cloud Functions
access to Google Cloud Storage bucket
Libraries: transformers, datasets, pandas, pandas-gbq, onnxruntime, onnx

Dataset

To make this blogpost both self-contained and complete, here’s how you can easily upload a portion of tweet_eval dataset into BigQuery. If you have some text already in BigQuery, you can skip this part.

python

import pandas as pd
from datasets import load_dataset

dataset = load_dataset('tweet_eval', 'sentiment')
validation_subset = dataset['validation'] # <-- use any split you want, here it's just for the demo purposes

df = pd.DataFrame(validation_subset)
df.to_gbq('onnx_demo.twitter_sentiment_validation', '<gcp-project-id>', if_exists='replace')

Preparing the model

The task is simple - find or train a model and convert in to ONNX format. I’m using the existing one: distilbert-base-uncased-finetuned-sst-2-english - it’s trained on 2 classes (positive and negative), not on tweets, but for the tutorial purposes that’s irrelevant - the goal is to deploy it and run the inference, so the steps for any other model will be the same.

⚠️ Limitation #1
BigQuery ML inference engine only supports ONNX models that are smaller than 450 MB. This is a serious limitation - to put this into context - deploying a popular sentiment-analysis model cardiffnlp/twitter-roberta-base-sentiment based on RoBERTa is NOT possible, because after the conversion to ONNX format, it’s size is 476MB.
State for 2023-05-14

To convert the model into ONNX format, you can just run the module provided by Huggingface’s transfomers library:

python

python -m transformers.onnx --model=distilbert-base-uncased-finetuned-sst-2-english --feature=sequence-classification onnx/ --opset 17

This command will output onnx/model.onnx file. Let’s run it locally first, to check if it’s working:

python

import onnxruntime as rt    
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')
sess = rt.InferenceSession("./onnx/model.onnx")
text = ['This is a positive tweet!', "Let's put some hate into this text", "Hello world!"]
inputs = tokenizer(text, return_tensors='np', padding=True)
output_name = sess.get_outputs()[0].name
output = sess.run([output_name], dict(inputs))
output[0].argmax(1)
# Output:
# array([1, 0, 1])

It works fine, but it will not work on BigQuery though…

⚠️ Limitation #2
BigQuery ML inference does not support multiple dynamic axes in ONNX models, the only dynamic axis can be the one responsible for batch size (which means - the first one).
State for 2023-05-14

Dynamic axes in ONNX allow to pass tensors of variable-lenght into the models - this is especially useful when you want to pass batches of various sizes into the model, instead of having them fixed to 1, 2,… etc. Models converted to ONNX with Huggingface’s transformers.onnx module have 2 dynamic axes - for NLP models it means: batch size and input sequence length. This is visible if you inspect the model:

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0]
# Output:
name: "input_ids"
type {
  tensor_type {
    elem_type: 7
    shape {
      dim {
        dim_param: "batch" # <--- first dynamic axis
      }
      dim {
        dim_param: "sequence" # <--- second dynamic axis
      }
    }
  }
}

Trying to deploy such model in BigQuery will result in the following error:

ONNX Model input ‘input_ids’ has unknown tensor dimension at index: 1, which is not supported.

How to fix it? - take 1

The model I’m using - DistilBERT - was trained with max_position_embeddings config set to 512, so let’s first try to set this value as a fixed size of the second axis in the ONNX’s models input (spoiler alert - it will not work).

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0].type.tensor_type.shape.dim[1].dim_value = 512 # <--- for input_ids
model.graph.input[1].type.tensor_type.shape.dim[1].dim_value = 512 # <--- for attention_mask
onnx.save(model, "./onnx/model_fixed.onnx")

Now, when I import the model into BigQuery it will still complain, this time - about something else:

ONNX model cannot be parsed within the memory limit; try reducing the model size

This happens, because ONNX needs to pre-allocate some memory for the input tensors, which also affect internal layer sizes, resulting in exhausted memory limit on the BigQuery side.

How to fix it? - take 2

I had to shrink the model inputs even further, down to 256 (which is still fine for short tweets)

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0].type.tensor_type.shape.dim[1].dim_value = 256 # <--- for input_ids
model.graph.input[1].type.tensor_type.shape.dim[1].dim_value = 256 # <--- for attention_mask
onnx.save(model, "./onnx/model_fixed_take_2.onnx")

Now it passes, BigQuery is able to create my model.

Creating BigQuery ONNX model

In order to import ONNX model into BigQuery inference engine, first upload the model (the fixed one :) ) to GCS:

bash

gsutil cp ./onnx/model_fixed_take_2.onnx gs://<your bucket>/models/distilbert-base-uncased-finetuned-sst-2-english.onnx

Then, open BigQuery UI and execute the following query:

sql

CREATE OR REPLACE MODEL
onnx_demo.sentiment_classifier
OPTIONS(MODEL_TYPE = 'ONNX', MODEL_PATH = "gs://<your bucket>/models/distilbert-base-uncased-finetuned-sst-2-english.onnx");

Once executed, you should see the model in the UI:

👀 Spoiler alert - you will have to fix one more thing in the ONNX model, follow the guide below.

Huggingface tokenizers in BigQuery

Wait… The model accepts input_ids and attention_mask, but our BigQuery table has text 😱 We need to tokenize the text first, before we pass it to the model, but tokenizers are Python code that cannot be easily converted to ONNX format. How to solve this problem? Use BigQuery Remote Functions!

Huggingface tokenizer as BigQuery Remote Function

BigQuery allows to create a UDF that can actually call either Cloud Functions or Cloud Run. Let’s leverage that. For more details also see my previous blog post here.

Full code of the Cloud Function:

python

import json
from transformers import AutoTokenizer
import functions_framework
import os

tokenizer_name = os.environ.get("TOKENIZER_NAME", "distilbert-base-uncased-finetuned-sst-2-english")
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)

def expand(input_dict):
    input_ids = input_dict["input_ids"]
    attention_mask = input_dict["attention_mask"]
    result = []
    for i in range(len(input_ids)):
        obj = {"input_ids": input_ids[i], "attention_mask": attention_mask[i]}
        result.append(obj)
    return result

@functions_framework.http
def tokenize_text(request):
    request_json = request.get_json()
    calls = request_json["calls"]
    texts = [row[0] for row in calls]
    tokenized = tokenizer(texts, truncation=True, max_length=256, padding="max_length")
    result = expand(tokenized)
    return json.dumps({"replies": [json.dumps(r) for r in result]}) # double serialization, because we need to return STR in Remote Function

First, I’m using functions_framework to be able to test my code locally. Secondly, I’m pre-loading distilbert-base-uncased-finetuned-sst-2-english on function start. Every call to the deployed function will be in BigQuery format, which looks like this:

python

{
  "request_id": "string",
  "caller": "string",
  "sessionUser": "string",
  "userDefinedContext": {},
  "calls": [
    ["values", "from", "row", 1],
    ["values", "from", "row", 2]
  ]
}

I’m invoking the tokenizer for the whole batch and then expand it from nested list into flat list of objects - each row will be in exact form that the ONNX model expects - it needs to have input_ids and attention_ids fields. Finally, the whole response is serialized, note the double serialization of JSON objects in each row. Right now BigQuery Remote Function does not support returning complex types, so I will just de-serialize this JSON afterwards in BigQuery SQL.

Make sure that you create requirements.txt file with all dependencies:

python

transformers~=4.29.1
functions-framework

Deploy the function using CLI:

bash

gcloud functions deploy tokenize_text --runtime python311 --trigger-http --min-instances 0 --no-allow-unauthenticated --region us-east1

Then, create connection between BigQuery and Cloud Functions and assing the Cloud Function Invoker role to the service account in this connection:

python

# 1. Create connection:
bq mk --connection --location=US --project_id=<project id> \
    --connection_type=CLOUD_RESOURCE huggingface-tokenizer
# 2. Copy the serviceAccountId from the results of:
bq show --format=prettyjson --connection <project id>.US.huggingface-tokenizer
# 3. Add Cloud Function Invoker role
gcloud functions add-iam-policy-binding tokenize_text --member="serviceAccount:bqcx-<redacted>@gcp-sa-bigquery-condel.iam.gserviceaccount.com" --role="roles/cloudfunctions.invoker" --region=us-east1

Once completed, add the remote function…

sql

CREATE OR REPLACE FUNCTION `onnx_demo`.tokenize(text STRING) RETURNS STRING
REMOTE WITH CONNECTION `<connection id>`
OPTIONS (
  endpoint = 'https://<cloud function url>.cloudfunctions.net/tokenize_text',
  max_batching_rows = 10 -- <-- adjust this depending on the memory limits of your Cloud Function
)

…and invoke it:

sql

select `onnx_demo`.tokenize(text) as t, text, label
from `onnx_demo.twitter_sentiment_validation` 
LIMIT 100

Result:

Great, we have deployed model and invoked the tokenizer, let’s put everything together!

DistilBERT inference on BigQuery

For clarity, I’ve split the inference code into 4 parts (CTEs):

Tokenizing the text
Transforming the tokenized JSONs into BigQuery arrays
Running the inference using the ONNX model
Parsing results to determine the text sentiment

Let’s start with #1, this query is simple - it’s just calling the tokenizer. Note that I’m explicitly limiting to 10 rows, see below why 👀

sql

with tokenized as (
  select `onnx_demo`.tokenize(text) as t, text, label
  from `onnx_demo.twitter_sentiment_validation` 
  LIMIT 10
)

Next part, #2 is parsing the JSONs from the tokenizer into BigQuery ARRAY<INT64> format:

sql

expanded as (
  select 
  (SELECT ARRAY_AGG(CAST(i AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.input_ids')) AS i) as input_ids, 
  (SELECT ARRAY_AGG(CAST(a AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.attention_mask')) AS a) as attention_mask, 
  text, label
  from tokenized
)

Calling the model (#3), while also passing the text and labels from the original table:

sql

predictions as (
  SELECT
    * EXCEPT(input_ids, attention_mask)
  FROM
    ML.PREDICT(MODEL `onnx_demo.sentiment_classifier`, (
      SELECT
        input_ids,
        attention_mask,
        text,
        label
      FROM
        expanded
      )
    )
)

Finally #4, parse the results (logits) from the model (if you have more than 2 classes in the model, you will effectively have to implement something similar to argmax to get the class predicted by the model).

sql

select text, (CASE WHEN logits[OFFSET(0)] > logits[OFFSET(1)] THEN 'negative' ELSE 'positive' END) as predicted, label, logits
from predictions

See the full query ⬇️

sql

with tokenized as (
  select `onnx_demo`.tokenize(text) as t, text, label
  from `onnx_demo.twitter_sentiment_validation` 
  LIMIT 10
), expanded as (
  select 
  (SELECT ARRAY_AGG(CAST(i AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.input_ids')) AS i) as input_ids, 
  (SELECT ARRAY_AGG(CAST(a AS INT64)) FROM UNNEST(JSON_EXTRACT_ARRAY(t, '$.attention_mask')) AS a) as attention_mask, 
  text, label
  from tokenized
), predictions as (
  SELECT
    * EXCEPT(input_ids, attention_mask)
  FROM
    ML.PREDICT(MODEL `onnx_demo.sentiment_classifier`, (
      SELECT
        input_ids,
        attention_mask,
        text,
        label
      FROM
        expanded
      )
    )
)
select text, (CASE WHEN logits[OFFSET(0)] > logits[OFFSET(1)] THEN 'negative' ELSE 'positive' END) as predicted, label, logits
from predictions

Result:

Scalling the inference

In the paragraph above, I’ve deliberately put a LIMIT 10 in the initial query, to limit the number of rows. Why? Try to run the query without the limit and see what happens:

Resources exceeded during query execution: UDF out of memory

⚠️ Limitation #3
BigQuery ML inference does not allow to set the batch size during model creation, which makes scalling up the inference challenging - you effectively don’t know how many rows will be send to the ONNX model at runtime, which might very often results in UDF out of memory exception.
State for 2023-05-14

Fortunately, there is a workaround for that. Remember, that original ONNX exported model had 2 dynamic axes - first one being batch_size and second one - sequence length. We’ve first reduced the second one to a fixed value of 256. Now it’s time to set the batch size to 1, which will force the BigQuery to only send one input example at a time to the model, making the inference possible for any number of rows. 💡 If you have any suggestion how it can be fixed in any other way, let me know in comments below.

python

import onnx
model = onnx.load('./onnx/model.onnx')
model.graph.input[0].type.tensor_type.shape.dim[0].dim_value = 1 # <--- force batch size to 1
model.graph.input[0].type.tensor_type.shape.dim[1].dim_value = 256
model.graph.input[1].type.tensor_type.shape.dim[0].dim_value = 1 # <--- force batch size to 1
model.graph.input[1].type.tensor_type.shape.dim[1].dim_value = 256
onnx.save(model, "./onnx/model_fixed_final.onnx")

Once you upload the model to GCS and re-create it using CREATE OR REPLACE MODEL, the inference should start to work at larger scale! 🎉

ONNX models vs Remote Functions

As you’ve seen, the ONNX feature is still in preview stage (Pre-GA), with some important limitations that you should be aware of. The biggest of them is the model size limit, making it impossible to deploy even medium size models, like base versions of BERT/RoBERTa. At the same time, a lot of data that customers store in BigQuery is tabular and those limits might be enough to deploy smaller models, e.g. TabNet model family. The API for ONNX models is also very simple, which makes it easy to use.

For NLP models that also require tokenization before doing the inference, Remote Functions are still a crucial building block. They allow to virtually run any Python code on the BigQuery side, meaning that you can use any Python library. The downside of this approach is that you need to write some code and deploy it separately as either Cloud Functions or Cloud Run.

Using Remote Functions for everything right now seems to have less limitations than ONNX models - you can deploy models on instances with up to 8 CPU and 32GB RAM (preview in Cloud Run), which will fit much larger models, with possibility of using quantized versions too.

If this post will get enough interest, I’ll be happy to prepare a side-by-side performance comparison of both approaches 📊.

Summary

I hope that this post helped you to deploy your own custom model in BigQuery ML. You also know the current limitations of ONNX models in BigQuery ML inference engine. Let me know what you think in the comments!

Additional links & resources

BigQuery Remote Functions documentation: https://cloud.google.com/bigquery/docs/remote-functions
BigQuery ML ONNX limitations: https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-onnx#limitations
BigQuery JSON functions: https://cloud.google.com/bigquery/docs/reference/standard-sql/json_functions

Deploy MLflow models on BigQuery

Marcin Zabłocki — Sun, 02 Apr 2023 01:00:00 +0000

Deploying machine learning models and making them easily accessible by other teams (especially data/business analysts) can be challenging. The landscape of MLOps tools in the recent months has exploded and opened a lot of opportunities for engineers (ML Engieeers / MLOps and you) to play with various systems and stich them together in creative and reliable way.

In this blogpost I will show you how you can easily deploy MLflow models to GCP Cloud Run service in a way that they could be consumed from BigQuery using SQL. I will achieve it by leveraging BigQuery Remote Functions feature.

💡 Recently (March 29, 2023) Google has announced a way to deploy models directly in BigQuery, I’ve also made a tutorial for this option here ➡️.
This blogpost shows an alternative option, which comes in handy, especially when you have custom logic / custom pre/post processing code in the models, that is not easily converted to ONNX format.

TL;DR

You will learn how to deploy MLflow models on Cloud Run & use them on BigQuery data using SQL. The whole project is available on GitHub (links below).

⚠️ What is not in the scope of this blogpost

I will not focus on the deployment of the MLflow instances here. Also, the model I will be deploying will be trained using TPOT’s AutoML for simplicity, as the Data Science part of the project is not important here.

What you will find below

Prerequisites

You will need:

Python (I’m using 3.9)
Docker
access to Google Cloud Platform (BigQuery & Cloud Run)
Artifact Registry for Docker / Google Container Registry available

Training a simple model with TPOT’s AutoML

As I’ve mentioned above, I will be not focusing on trainign the best-in-class model for the dataset, so for the simplicity, I’m using AutoML capabilities provided by the TPOT library.

The library has really simple API and the general workflow with it looks like this:

Load the data.

Create a classification or regression task

Set AutoML parameters, especially: number of generations, population size and paralleization level (they affect training time, usually - the longer = the better model)

Export the training pipeline - TPOT exports the scratch version of the training pipeline, that you can modify or copy to your existing project.

Modify the scratch version of the pipeline to load the data, train the model and save it somewhere.

For the save it somewhere part I will be using MLflow as it handles model serialization well: it not only saves (pickles) the trained model but also captures input data schema and input examples - this is really important from both usability and MLOps perspective, as you always know what to “send” to the model during inference.

In this blogpost, I will be using California Housing dataset (available directly from Scikit Learn).

Open requirements.txt

python

cloudpickle==2.2.1
fastapi==0.92.0
uvicorn==0.21.0
mlflow
scikit-learn==1.2.1
pandas==1.5.3
tpot==0.11.7
numpy==1.23.5
scipy==1.10.1

python

from functools import partial, update_wrapper
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from tpot import TPOTRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_california_housing


def main():
    # split into train and test
    dataset = fetch_california_housing(as_frame=True)
    dataset.target = dataset.target * 100000
    X_train, X_test, y_train, y_test = train_test_split(
        dataset.data.astype(float),
        dataset.target,
        train_size=0.7,
        test_size=0.3,
    )

    for ds, name in zip(
        (X_train, X_test, y_train, y_test), ("X_train", "X_test", "y_train", "y_test")
    ):
        ds.to_csv(f"{name}.csv", index=False)

    # create and fit TPOT
    tpot = TPOTRegressor(
        generations=10,
        population_size=50,
        verbosity=2,
        random_state=666,
        n_jobs=-1,
        scoring="neg_mean_absolute_error",
        config_dict="TPOT light",
    )
    tpot.fit(X_train, y_train)
    # calculate mean_absolute_error, means_squared_error, r2_score in a loop
    predictions = tpot.predict(X_test)
    for metric in [
        mean_absolute_error,
        update_wrapper(partial(mean_squared_error, squared=False), mean_squared_error),
        r2_score,
    ]:
        print(f"{metric.__name__}: {metric(y_test, predictions)}")
    tpot.export("tpot_pipeline_TO_EDIT.py")


if __name__ == "__main__":
    main()

After running the code above, you will see tpot_pipeline_TO_EDIT.py file in your working directory. It contains the best model that TPOT’s AutoML has found.

I’ve copied this file as tpot_pipeline.py and edited it by adding:

loading of my data
saving the model in the MLflow format.

python

import mlflow
import pandas as pd
from mlflow.models import infer_signature
from sklearn.feature_selection import VarianceThreshold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler
from sklearn.svm import LinearSVR
from sklearn.tree import DecisionTreeRegressor
from tpot.builtins import StackingEstimator
from tpot.export_utils import set_param_recursive

X_train = pd.read_csv("X_train.csv")
y_train = pd.read_csv("y_train.csv")


with mlflow.start_run() as run:
    # Average CV score on the training set was: -39252.13134203072
    exported_pipeline = make_pipeline(
        RobustScaler(),
        VarianceThreshold(threshold=0.0001),
        StackingEstimator(
            estimator=LinearSVR(
                C=10.0,
                dual=True,
                epsilon=0.1,
                loss="squared_epsilon_insensitive",
                tol=0.001,
            )
        ),
        DecisionTreeRegressor(max_depth=10, min_samples_leaf=14, min_samples_split=4),
    )
    # Fix random state for all the steps in exported pipeline
    set_param_recursive(exported_pipeline.steps, "random_state", 666)

    exported_pipeline.fit(X_train, y_train)

    test_df = pd.read_csv("X_test.csv")
    mlflow.sklearn.save_model(
        exported_pipeline,
        "model",
        signature=infer_signature(X_train),
        input_example=test_df.sample(n=7, random_state=666),
    )
    print("Done :)")

Once you run this script, the “best” model will be trained and saved on disk in the model folder. MLflow’s model format is described here.

Creating BigQuery Remote Function with Fast API

First, let me explain how the BigQuery Remote Functions work: you create either a Cloud Function or Cloud Run endpoint, which needs to implement a contract defined in the documentation. Then, you create a Cloud resource connection in BigQuery - this will create a Service Account, that BQ will use to access the Cloud Function / Cloud Run service. After setting appropriate IAM to this service account, you create a definition for the remote function within BQ, specifying its signature, endpoint URL and options. Once everything is set, you will be able to run the Remote Function just like any other function in BigQuery - directly from SQL!

BigQuery Remote Function contract

BigQuery has a well defined format of the requests its sending to any Remote Function. An example request looks like this:

javascript

{
 "requestId": "124ab1c",
 "caller": "//bigquery.googleapis.com/projects/myproject/jobs/myproject:US.bquxjob_5b4c112c_17961fafeaf",
 "sessionUser": "test-user@test-company.com",
 "userDefinedContext": {
  "key1": "value1",
  "key2": "v2" // <--- any metadata you like to attach (it's defined when you register the function in BQ)
 },
 "calls": [
  [null, 1, "", "abc"], // <--- first row from the request
  ["abc", "9007199254740993", null, null] // <--- second row from the request
 ]
}

In order to implement a FastAPI endpoint accepting requests in the format shown above, let’s create a pydantic dataclass for it:

python

from typing import Dict, List, Any, Optional
from pydantic import BaseModel

class BigQueryUDFRequest(BaseModel):
    request_id: Optional[str]
    caller: str
    sessionUser: str
    userDefinedContext: Optional[Dict]
    calls: List[List[Any]] = []

The response contract is simpler, it only needs a single replies field, as shown below.

python

class BigQueryUDFResponse(BaseModel):
    replies: List[Any]

Last thing - BigQuery Remote Functions only support root level endpoints - that means, that the endpoint in the FastAPI must accept POST requests on the / (root) path. No nested paths are allowed at the moment of writing this blogpost.

Remote Function endpoint

Definition of an endpoint and the FastAPI appplication looks like this:

python

import mlflow.sklearn
import pandas as pd
from fastapi import FastAPI
from api.models import BigQueryUDFRequest, BigQueryUDFResponse, AppContext

app = FastAPI()
ctx = AppContext()

@app.on_event("startup")
def load_model():
    # Load your MLflow model here
    ctx.model = mlflow.sklearn.load_model("model")

@app.post("/", response_model=BigQueryUDFResponse)
def udf(request: BigQueryUDFRequest):
    df = pd.DataFrame(request.calls)
    # Optionally - parse the request, do some logging etc
    predictions = ctx.model.predict(df)
    return BigQueryUDFResponse(replies=predictions.tolist())

The AppContext is defined like this:

python

@dataclass
class AppContext:
    model: Any = None

Testing out the endpoint locally

Once you run the app locally, you can either open http://localhost:8000/docs and send requests from Swagger UI, or use curl. For my AutoML model, the request looks like this:

bash

curl -X 'POST' \
  'http://localhost:8000/' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "request_id": "string",
  "caller": "string",
  "sessionUser": "string",
  "userDefinedContext": {},
  "calls": [[3.1779, 16.0, 4.636165577342048, 0.9607843137254902, 1860.0, 4.052287581699346, 38.04, -121.29], 
            [4.1364, 24.0, 23.54736842105263, 5.168421052631579, 264.0, 2.778947368421053, 39.27, -120.04]]
}'

As you can see, batch requests are possible out-of-the-box.

Deploying Remote Function to Cloud Run

Building a Docker container

First, a docker container with our custom function needs to be build.

For tutorial purposes, the MLflow model will be copied into the container itself. In production scenarios, you will most likely load the model on the fly from GCS during the container startup process.

An ENTRYPOINT for Cloud Run and Fast API can look like this (save this as run.sh). It’s important to dyunamically use the $PORT environment variable, as it will be set by the Cloud Run’s runtime.

python

#!/bin/bash
set -e
exec uvicorn bq_api:app --proxy-headers --host 0.0.0.0 --port $PORT

… and the Dockerfile itself:

python

FROM python:3.9.16-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt --no-cache-dir
COPY . .
RUN chmod +x ./run.sh
ENV PORT 8000
ENTRYPOINT ["./run.sh"]

Build the docker image.

python

export IMAGE="us-east1-docker.pkg.dev/<full path to artifact registry>/mlflow-bq-example:20220401-1942"

# On Macs with M1/M2 processor:
docker buildx build --platform=linux/amd64 -t $IMAGE .

# On Linux/other:
docker build -t $IMAGE .

Now, push the image to the Artifact Registry / Container Registry (docker push $IMAGE).

Create Cloud Run service

With GCLOUD CLI, deployment to Cloud Run is a one-liner. The command provides sensible defaults for the tutorial purposes.

bash

gcloud run deploy mlflow-bigquery \
    --cpu 1 --memory 1Gi --image=$IMAGE \
    --region=us-east1 --no-allow-unauthenticated

after a few seconds, the deployment should finish:

bash

Deploying container to Cloud Run service [mlflow-bigquery] in project [<project>] region [us-east1]
✓ Deploying new service... Done.                                                                                                                                                                        
  ✓ Creating Revision...                                                                                                                                                                                
  ✓ Routing traffic...                                                                                                                                                                                  
Done.                                                                                                                                                                                                   
Service [mlflow-bigquery] revision [mlflow-bigquery-00001-yav] has been deployed and is serving 100 percent of traffic.
Service URL: https://mlflow-bigquery-xyz123-ue.a.run.app

Thanks to the --no-allow-unauthenticated, the Cloud Run service will be protected from unauthorized access by default. In order to send an authorized requests to the Cloud Run service, use the following command:

bash

curl -X 'POST' \
  'https://mlflow-bigquery-xyz123-ue.a.run.app' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -d '{
  "request_id": "string",
  "caller": "string",
  "sessionUser": "string",
  "userDefinedContext": {},
  "calls": [[3.1779, 16.0, 4.636165577342048, 0.9607843137254902, 1860.0, 4.052287581699346, 38.04, -121.29], 
            [4.1364, 24.0, 23.54736842105263, 5.168421052631579, 264.0, 2.778947368421053, 39.27, -120.04]]
}'

result should be exactly the same as before.

Connecting from BigQuery to Cloud Run

⚠️ Before proceeding, make sure that you have BigQuery Connection API enabled in your GCP project.

You can either configure the connection from UI or use command line.

(option 1) From UI

Click Add

Click Connections to external data sources
Fill in the form, in the Connection type list, use BigLake and remote functions (Cloud Resource).

Click Create connection button.
In the left pane, open External connections and view the created connection details. Copy Service account id for later use.

(option 2) From CLI

bash

bq mk --connection --location=US --project_id=<project id> \
    --connection_type=CLOUD_RESOURCE mlflow-cloudrun-cli

javascript

bq show --format=prettyjson --connection <project id>.US.mlflow-cloudrun-cli
{
  "cloudResource": {
    "serviceAccountId": "bqcx-<redacted>@gcp-sa-bigquery-condel.iam.gserviceaccount.com"
  },
  "creationTime": "1680373492742",
  "lastModifiedTime": "1680373492742",
  "name": "projects/<redacted>/locations/us/connections/mlflow-cloudrun-cli"
}

Once the External connection and service account id is created, we need to allow this service to access Cloud Run.

bash

gcloud run services add-iam-policy-binding mlflow-bigquery \
  --member='serviceAccount:bqcx-<redacted>@gcp-sa-bigquery-condel.iam.gserviceaccount.com' \
  --role='roles/run.invoker' --region=us-east1

Running the inference from BigQuery

Now it’s time to create the actual function to invoke from BigQuery. Open query editor and create function definition similar to the one below. Make sure that you set the number of parameters right.

💡 The max_batching_rows = 128 parameter is really important as it affects performance of the inference - it effectively tells BigQuery, how many rows of the input table to send to Cloud Run in a single request. Setting the max_batching_rows as well as the concurrency in the Cloud Run (max requests per container) allows to fine-tune the inference performance.

sql

CREATE OR REPLACE FUNCTION `<project id>.mlflow_model_demo`.predict(
  MedInc FLOAT64,
  HouseAge FLOAT64,
  AveRooms FLOAT64,
  AveBedrms FLOAT64,
  Population FLOAT64,
  AveOccup FLOAT64,
  Latitude FLOAT64,
  Longitude FLOAT64
) RETURNS FLOAT64
REMOTE WITH CONNECTION `<project id>.US.mlflow-bigquery-cloudrun`
OPTIONS (
  endpoint = 'https://mlflow-bigquery-xyz123-ue.a.run.app'
  max_batching_rows = 128
)

The above statement will create a new custom function named predict in the mlflow_model_demo dataset. After successful run, the following message should display in the query results panel:

bash

This statement created a new function named <project id>.mlflow_model_demo.predict.

Once the function is created, let’s use it!

sql

SELECT *,
`<project id>.mlflow_model_demo`.predict(
    MedInc,
    HouseAge,
    AveRooms,
    AveBedrms,
    Population,
    AveOccup,
    Latitude,
    Longitude
) as price,
 FROM `<project id>.mlflow_model_demo.california_housing_test` LIMIT 1000

after a few seconds, results will display:

In the Cloud Run dashboard, you will be able to observe the performance metrics - they will be an useful input for setting the max_batching_rows in BQ and concurrency in Cloud Run to achieve optimal ML model inference performance in this setup.

💡Pro-tips

Here’s a bunch of pr0-tips useful while working with this setup:

MLflow model saving

When you save the model in MLflow, use input_example parameter of save_model- it will give you a JSON with exact data shape and types that your model accepts.
Also in MLflow, use mlflow.models.signature.infer_signature and signature param of save_model to capture metadata about data types - also useful for future debugging.

Multiple Remote Functions in a single endpoint

You actually CAN have multiple BigQuery Remote Functions deployed in a single Cloud Run Service. In order to do this, you need to specify user_defined_context during function create and handle routing in the root endpoint on your own.

See the code

sql

CREATE OR REPLACE FUNCTION `<project id>.mlflow_model_demo`.other_function(
  col1 FLOAT64,
  col1 FLOAT64,
) RETURNS FLOAT64
REMOTE WITH CONNECTION `<project id>.US.mlflow-bigquery-cloudrun`
OPTIONS (
  user_defined_context = [("function", "other-function")]
)

Then, in the function handler in FastAPI do the routing on the contents of request.userDefinedContext dictionary

python

@app.post("/", response_model=BigQueryUDFResponse)
def udf(request: BigQueryUDFRequest):
    df = pd.DataFrame(request.calls)
    if request.userDefinedContext["function"] == "predict":
        results = ctx.model.predict(df)
    elif request.userDefinedContext["function"] == "other-function":
        results = call.other.function()
    return BigQueryUDFResponse(replies=results.tolist())

Summary

I hope that this post helped you to deploy any custom MLflow model to Cloud Run and use it directly from BigQuery just like any other SQL function.

Additional links & resources

GitHub repo with full project - https://github.com/marrrcin/mlflow-bigquery-remote-function
Deploying secure MLflow on App Engine: https://getindata.com/blog/deploying-mlflow-google-cloud-platform-using-app-engine/
Deploying secure MLflow on Cloud Run: https://getindata.com/blog/deploying-serverless-mlflow-google-cloud-platform-using-cloud-run/

Understanding & implementing SimCLR in PyTorch - an ELI5 guide

Marcin Zabłocki — Mon, 27 Jul 2020 14:57:48 +0200

Transfer learning and pre-training schemas for both NLP and Computer Vision have gained a lot of attention in the last months. Research showed that carefully designed unsupervised/self-supervised training can produce high quality base models and embeddings that greatly decrease the amount of data needed to obtain good classification models downstream. This approach becomes more and more important as the companies collect a lot of data from which only a fraction can be labelled by humans - either due to large cost of labelling process or due to some time constraints.

Here I explore SimCLR pre-training framework proposed by Google in this arxiv paper. I will explain the SimCLR and its contrastive loss function step by step, starting from naive implementation in PyTorch, followed by faster, vectorized one. Then I will show how to use SimCLR’s pre-training routine to first build image embeddings using EfficientNet network architecture and finally I will show how to build a classifier on top of it.

TL;DR

This post covers:

understanding the SimCLR framework with code samples in PyTorch
from scratch explanation & implementation of SimCLR’s loss function (NT-Xent) in PyTorch
pre-training image embeddings using EfficientNet architecture
training classifier by using transfer learning from the pre-trained embeddings

Understanding SimCLR framework

In general, SimCLR is a simple framework for contrastive learning of visual representations. It’s not any new framework for deep learning, it’s a set of fixed steps that one should follow in order to train image embeddings of good quality. I drew a schema which explains the flow and the whole representation learning process (click to zoom).

The flow is as follows (from left to right):

Take an input image
Prepare 2 random augmentations on the image, including: rotations, hue/saturation/brightness changes, zooming, cropping, etc. The range of augmentations as well as analysis which ones work best are discussed in detail in the paper.
Run a deep neural network (preferably convolutional one, like ResNet50) to obtain image representations (embeddings) for those augmented images.
Run a small fully connected linear neural network to project embeddings into another vector space.
Calculate contrastive loss and run backpropagation through both networks. Contrastive loss decreases when projections coming from the same image are similar. The similarity between projections can be arbitrary, here I will use cosine similarity, same as in the paper.

Contrastive loss function

Theory behind contrastive loss function

One can reason about contrastive loss function form two angles:

Contrastive loss decreases when projections of augmented images coming from the same input image are similar.
For two augmented images: (i), (j) (coming from the same input image - I will call them “positive” pair later on), the contrastive loss for (i) tries to identify (j) among other images (“negative” examples) that are in the same batch .

Formal definition of the loss for pair of positive examples (i) and (j) defined as:

The final loss is an arithmetic mean of the losses for all positive pairs in the batch:

(keep in mind that the indexing in l(2k-1, 2k) + l(2k, 2k-1) is purely dependent on how you implement the loss - I find it easier to understand when I reason about them as l(i,j) + l(j,i)).

Contrastive loss function - implementation in PyTorch, ELI5 version

It’s much easier to implement the loss function without vectorization first and then follow up with the vectorization phase.

python

import torch
from torch import nn
import torch.nn.functional as F

python

class ContrastiveLossELI5(nn.Module):
    def __init__(self, batch_size, temperature=0.5, verbose=True):
        super().__init__()
        self.batch_size = batch_size
        self.register_buffer("temperature", torch.tensor(temperature))
        self.verbose = verbose
            
    def forward(self, emb_i, emb_j):
        """
        emb_i and emb_j are batches of embeddings, where corresponding indices are pairs
        z_i, z_j as per SimCLR paper
        """
        z_i = F.normalize(emb_i, dim=1)
        z_j = F.normalize(emb_j, dim=1)

        representations = torch.cat([z_i, z_j], dim=0)
        similarity_matrix = F.cosine_similarity(representations.unsqueeze(1), representations.unsqueeze(0), dim=2)
        if self.verbose: print("Similarity matrix\n", similarity_matrix, "\n")
            
        def l_ij(i, j):
            z_i_, z_j_ = representations[i], representations[j]
            sim_i_j = similarity_matrix[i, j]
            if self.verbose: print(f"sim({i}, {j})={sim_i_j}")
                
            numerator = torch.exp(sim_i_j / self.temperature)
            one_for_not_i = torch.ones((2 * self.batch_size, )).to(emb_i.device).scatter_(0, torch.tensor([i]), 0.0)
            if self.verbose: print(f"1{{k!={i}}}",one_for_not_i)
            
            denominator = torch.sum(
                one_for_not_i * torch.exp(similarity_matrix[i, :] / self.temperature)
            )    
            if self.verbose: print("Denominator", denominator)
                
            loss_ij = -torch.log(numerator / denominator)
            if self.verbose: print(f"loss({i},{j})={loss_ij}\n")
                
            return loss_ij.squeeze(0)

        N = self.batch_size
        loss = 0.0
        for k in range(0, N):
            loss += l_ij(k, k + N) + l_ij(k + N, k)
        return 1.0 / (2*N) * loss

Explanation

Contrastive loss needs to know the batch size and temperature (scaling) parameter. You can find details about setting the optimal temperature parameter in the paper.

My implementation of the forward of the contrastive loss takes two parameters. First one will be a batch projection of images after first augmentation, the second will be a batch projection of images after second augmentation.

Projections needs to be normalized first, hence:

python

z_i = F.normalize(emb_i, dim=1)
z_j = F.normalize(emb_j, dim=1)

All representations are concatenated together in order to efficiently calculate cosine similarities between each image pair.

python

representations = torch.cat([z_i, z_j], dim=0)
similarity_matrix = F.cosine_similarity(representations.unsqueeze(1), representations.unsqueeze(0), dim=2)

Next is the naive implementation of l(i,j) for clarity and easiness of understanding. The code bellow almost directly implements the equation:

python

def l_ij(i, j):
    z_i_, z_j_ = representations[i], representations[j]
    sim_i_j = similarity_matrix[i, j]
    numerator = torch.exp(sim_i_j / self.temperature)
    one_for_not_i = torch.ones((2 * self.batch_size, )).to(emb_i.device).scatter_(0, torch.tensor([i]), 0.0)
    denominator = torch.sum(
        one_for_not_i * torch.exp(similarity_matrix[i, :] / self.temperature)
    )

    loss_ij = -torch.log(numerator / denominator)
    return loss_ij.squeeze(0)

Then, the final loss for the batch is computed as an arithmetic mean of all combinations of positive examples:

python

N = self.batch_size
loss = 0.0
for k in range(0, N):
    loss += l_ij(k, k + N) + l_ij(k + N, k)
return 1.0 / (2*N) * loss

Now, let’s run it with verbose mode to see what’s inside.

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]])
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]])
loss_eli5 = ContrastiveLossELI5(batch_size=3, temperature=1.0, verbose=True)
loss_eli5(I, J)

bash

Similarity matrix
    tensor([[ 1.0000, -0.1240,  0.9648,  0.8944, -0.0948,  0.9679],
        [-0.1240,  1.0000, -0.3807,  0.3328,  0.9996, -0.3694],
        [ 0.9648, -0.3807,  1.0000,  0.7452, -0.3534,  0.9999],
        [ 0.8944,  0.3328,  0.7452,  1.0000,  0.3604,  0.7533],
        [-0.0948,  0.9996, -0.3534,  0.3604,  1.0000, -0.3419],
        [ 0.9679, -0.3694,  0.9999,  0.7533, -0.3419,  1.0000]]) 

sim(0, 3)=0.8944272398948669
1{k!=0} tensor([0., 1., 1., 1., 1., 1.])
Denominator tensor(9.4954)
loss(0,3)=1.3563847541809082

sim(3, 0)=0.8944272398948669
1{k!=3} tensor([1., 1., 1., 0., 1., 1.])
Denominator tensor(9.5058)
loss(3,0)=1.357473373413086

sim(1, 4)=0.9995677471160889
1{k!=1} tensor([1., 0., 1., 1., 1., 1.])
Denominator tensor(6.3699)
loss(1,4)=0.8520082831382751

sim(4, 1)=0.9995677471160889
1{k!=4} tensor([1., 1., 1., 1., 0., 1.])
Denominator tensor(6.4733)
loss(4,1)=0.8681114912033081

sim(2, 5)=0.9999250769615173
1{k!=2} tensor([1., 1., 0., 1., 1., 1.])
Denominator tensor(8.8348)
loss(2,5)=1.1787779331207275

sim(5, 2)=0.9999250769615173
1{k!=5} tensor([1., 1., 1., 1., 1., 0.])
Denominator tensor(8.8762)
loss(5,2)=1.1834462881088257

tensor(1.1327)

A few things happened there, but by going back and forward between the verbose logs and the equation, everything should become clear. The indexing jumps by batch size (first l(0,3), l(3,0) then l(1,4), l(4,1) because of the way the similarity matrix was constructed. First row of the similarity_matrix is:

python

[ 1.0000, -0.1240,  0.9648,  0.8944, -0.0948,  0.9679]

Remember the input:

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]])
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]])

Now:

1.0000 is the cosine similarity between I[0] and I[0] ([1.0, 2.0] and [1.0, 2.0])

-0.1240 is the cosine similarity between I[0] and I[1] ([1.0, 2.0] and [3.0, -2.0])

-0.0948 is the cosine similarity between I[0] and J[2] ([1.0, 2.0] and [2.8, -1.75])

… and so on.

Let’s see if the loss decreases if the similarity between first image projection increases:

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]])
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]])
J = torch.tensor([[1.0, 1.75], [2.8, -1.75], [1.0, 4.7]]) # note the change
ContrastiveLossELI5(3, 1.0, verbose=False)(I, J)

bash

tensor(1.0996)

Indeed the loss decreases! Now I will follow up with the vectorized implementation.

Contrastive loss function - implementation in PyTorch, vectorized version

Performance of naive implementation is really poor (mostly due to manual loop), see the results:

python

contrastive_loss_eli5 = ContrastiveLossELI5(3, 1.0, verbose=False)
I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]], requires_grad=True)
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]], requires_grad=True)

python

%%timeit
contrastive_loss_eli5(I, J)
838 µs ± 23.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Once I understood the internals of the loss, it’s easy to vectorize it and remove the manual loop:

python

class ContrastiveLoss(nn.Module):
    def __init__(self, batch_size, temperature=0.5):
        super().__init__()
        self.batch_size = batch_size
        self.register_buffer("temperature", torch.tensor(temperature))
        self.register_buffer("negatives_mask", (~torch.eye(batch_size * 2, batch_size * 2, dtype=bool)).float())
            
    def forward(self, emb_i, emb_j):
        """
        emb_i and emb_j are batches of embeddings, where corresponding indices are pairs
        z_i, z_j as per SimCLR paper
        """
        z_i = F.normalize(emb_i, dim=1)
        z_j = F.normalize(emb_j, dim=1)

        representations = torch.cat([z_i, z_j], dim=0)
        similarity_matrix = F.cosine_similarity(representations.unsqueeze(1), representations.unsqueeze(0), dim=2)
        
        sim_ij = torch.diag(similarity_matrix, self.batch_size)
        sim_ji = torch.diag(similarity_matrix, -self.batch_size)
        positives = torch.cat([sim_ij, sim_ji], dim=0)
        
        nominator = torch.exp(positives / self.temperature)
        denominator = self.negatives_mask * torch.exp(similarity_matrix / self.temperature)
    
        loss_partial = -torch.log(nominator / torch.sum(denominator, dim=1))
        loss = torch.sum(loss_partial) / (2 * self.batch_size)
        return loss

python

contrastive_loss = ContrastiveLoss(3, 1.0)
contrastive_loss(I, J).item() - contrastive_loss_eli5(I, J).item()
0.0

The difference should be zero or close to zero (< 1e-6 due to fp arithmetics). Performance comparison:

python

I = torch.tensor([[1.0, 2.0], [3.0, -2.0], [1.0, 5.0]], requires_grad=True)
J = torch.tensor([[1.0, 0.75], [2.8, -1.75], [1.0, 4.7]], requires_grad=True)

python

%%timeit
contrastive_loss_eli5(I, J)
918 µs ± 60.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

python

%%timeit
contrastive_loss(I, J)
272 µs ± 9.18 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Almost 4x improvement, it works.

Pre-training image embeddings using SimCLR with EfficientNet

Once the loss function is established and understood it’s time to make a good use of it. I will pre-train image embeddings using EfficientNet architecture, following the SimCLR framework. For convenience, I’ve implemented a few utility functions and classes that I will explain briefly below. The training code is structured using PyTorch-Lightning.

I will use a great EfficientNet [https://arxiv.org/abs/1905.11946] implementation by Luke Melas-Kyriazi from GitHub, already pre-trained on ImageNet (transfer learning inception!). The dataset I choose is STL10 (from torchvision) as it contains both training and unlabeled splits for unsupervised / self-supervised learning tasks.

My goal here is to demonstrate the whole SimCLR flow from start to the end. I had no intent to reach new SOTA with the presented configuration.

Utility functions for image augmentations

Training with SimCLR produces good image embeddings that are not affected by image transformations - this is because during training, various data augmentations are done to force the network to understand the contents of the images regardless of i.e color of the image or the position where the object in the image is placed. SimCLR’s authors say that composition of data augmentations plays a critical role in defining effective predictive tasks and also contrastive learning needs stronger data augmentation than supervised learning. To sum this up: when pre-training the image embeddings it’s good to make this task difficult for the network to learn by strongly augmenting the images in order to generalize better afterwards.

I strongly advise to read the SimCLR’s paper and it’s appendix as they did ablation studies on which data augmentations bring the best effects on the embeddings.

To keep this blog-post simple to go through, I will mostly use built-in Torchvision’s data augmentations , with one additional one - random resized rotation.

python

def random_rotate(image):
    if random.random() > 0.5:
        return tvf.rotate(image, angle=random.choice((0, 90, 180, 270)))
    return image

python

class ResizedRotation():
    def __init__(self, angle, output_size=(96, 96)):
        self.angle = angle
        self.output_size = output_size
        
    def angle_to_rad(self, ang): return np.pi * ang / 180.0
        
    def __call__(self, image):
        w, h = image.size
        new_h = int(np.abs(w * np.sin(self.angle_to_rad(90 - self.angle))) + np.abs(h * np.sin(self.angle_to_rad(self.angle))))
        new_w = int(np.abs(h * np.sin(self.angle_to_rad(90 - self.angle))) + np.abs(w * np.sin(self.angle_to_rad(self.angle))))
        img = tvf.resize(image, (new_w, new_h))
        img = tvf.rotate(img, self.angle)
        img = tvf.center_crop(img, self.output_size)
        return img

python

class WrapWithRandomParams():
    def __init__(self, constructor, ranges):
        self.constructor = constructor
        self.ranges = ranges
    
    def __call__(self, image):
        randoms = [float(np.random.uniform(low, high)) for _, (low, high) in zip(range(len(self.ranges)), self.ranges)]
        return self.constructor(*randoms)(image)

python

from torchvision.datasets import STL10
import torchvision.transforms.functional as tvf
from torchvision import transforms
import numpy as np

A brief look on the transformation results:

python

stl10_unlabeled = STL10(".", split="unlabeled", download=True)

python

idx = 123

python

random_resized_rotation = WrapWithRandomParams(lambda angle: ResizedRotation(angle), [(0.0, 360.0)])

python

random_resized_rotation(tvf.resize(stl10_unlabeled[idx][0], (96, 96)))

Automatic data augmentation wrapper

Here I’ve also implemented a utility dataset wrapper that automatically applies random data augmentations every time when the image is retrieved. It can be easily used with any image dataset as long as it follows the simple interface of returning tuple with (PIL Image, anything). This wrapper can be set to return a deterministic transformation with the debug flag set to True. Note that there is a preprocess step that applies ImageNet-originated data standardization as I’m using already pre-trained EfficientNet.

python

from torch.utils.data import Dataset, DataLoader, SubsetRandomSampler, SequentialSampler
import random

python

class PretrainingDatasetWrapper(Dataset):
    def __init__(self, ds: Dataset, target_size=(96, 96), debug=False):
        super().__init__()
        self.ds = ds
        self.debug = debug
        self.target_size = target_size
        if debug:
            print("DATASET IN DEBUG MODE")
        
        # I will be using network pre-trained on ImageNet first, which uses this normalization.
        # Remove this, if you're training from scratch or apply different transformations accordingly
        self.preprocess = transforms.Compose([
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ])
        
        random_resized_rotation = WrapWithRandomParams(lambda angle: ResizedRotation(angle, target_size), [(0.0, 360.0)])
        self.randomize = transforms.Compose([
            transforms.RandomResizedCrop(target_size, scale=(1/3, 1.0), ratio=(0.3, 2.0)),
            transforms.RandomChoice([
                transforms.RandomHorizontalFlip(p=0.5),
                transforms.Lambda(random_rotate)
            ]),
            transforms.RandomApply([
                random_resized_rotation
            ], p=0.33),
            transforms.RandomApply([
                transforms.ColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0.2)
            ], p=0.8),
            transforms.RandomGrayscale(p=0.2)
        ])
    
    def __len__(self): return len(self.ds)
    
    def __getitem_internal__(self, idx, preprocess=True):
        this_image_raw, _ = self.ds[idx]
        
        if self.debug:
            random.seed(idx)
            t1 = self.randomize(this_image_raw)
            random.seed(idx + 1)
            t2 = self.randomize(this_image_raw)
        else:
            t1 = self.randomize(this_image_raw)
            t2 = self.randomize(this_image_raw)
        
        if preprocess:
            t1 = self.preprocess(t1)
            t2 = self.preprocess(t2)
        else:
            t1 = transforms.ToTensor()(t1)
            t2 = transforms.ToTensor()(t2)

        return (t1, t2), torch.tensor(0)

    def __getitem__(self, idx):
        return self.__getitem_internal__(idx, True)
    
    def raw(self, idx):
        return self.__getitem_internal__(idx, False)

python

ds = PretrainingDatasetWrapper(stl10_unlabeled, debug=False)

python

tvf.to_pil_image(ds[idx][0][0])

python

tvf.to_pil_image(ds[idx][0][1])

python

tvf.to_pil_image(ds.raw(idx)[0][1])

python

tvf.to_pil_image(ds.raw(idx)[0][0])

SimCLR neural network for embeddings

Here I define the ImageEmbedding neural network which is based on EfficientNet-b0 architecture. I swap out the last layer of pre-trained EfficientNet with identity function and add projection for image embeddings on top of it (following the SimCLR paper) with Linear-ReLU-Linear layers. It was shown in the paper that the non-linear projection head (i.e Linear-ReLU-Linear) improves the quality of the embeddings.

python

from efficientnet_pytorch import EfficientNet

python

class ImageEmbedding(nn.Module):       
    class Identity(nn.Module):
        def __init__(self): super().__init__()

        def forward(self, x):
            return x
    
        
    def __init__(self, embedding_size=1024):
        super().__init__()
        
        base_model = EfficientNet.from_pretrained("efficientnet-b0")
        internal_embedding_size = base_model._fc.in_features
        base_model._fc = ImageEmbedding.Identity()
        
        self.embedding = base_model
        
        self.projection = nn.Sequential(
            nn.Linear(in_features=internal_embedding_size, out_features=embedding_size),
            nn.ReLU(),
            nn.Linear(in_features=embedding_size, out_features=embedding_size)
        )

    def calculate_embedding(self, image):
        return self.embedding(image)

    def forward(self, X):
        image = X
        embedding = self.calculate_embedding(image)
        projection = self.projection(embedding)
        return embedding, projection

Next is the implementation of PyTorch-Lightning-based training module that orchestrates everything together:

hyper-parameters handling
SimCLR ImageEmbedding network
STL10 dataset
optimizer
forward step

As the PretrainingDatasetWrapper I’ve implemented returns tuple of: (Image1, Image2), dummy class, the forward step for this module is straightforward - it needs to produce two batches of embeddings and calculate contrastive loss function:

python

(X, Y), y = batch
embX, projectionX = self.forward(X)
embY, projectionY = self.forward(Y)
loss = self.loss(projectionX, projectionY)

python

from torch.multiprocessing import cpu_count
from torch.optim import RMSprop
import pytorch_lightning as pl

python

class ImageEmbeddingModule(pl.LightningModule):
    def __init__(self, hparams):
        hparams = Namespace(**hparams) if isinstance(hparams, dict) else hparams
        super().__init__()
        self.hparams = hparams
        self.model = ImageEmbedding()
        self.loss = ContrastiveLoss(hparams.batch_size)
    
    def total_steps(self):
        return len(self.train_dataloader()) // self.hparams.epochs
    
    def train_dataloader(self):
        return DataLoader(PretrainingDatasetWrapper(stl10_unlabeled, 
                                             debug=getattr(self.hparams, "debug", False)),
                          batch_size=self.hparams.batch_size, 
                          num_workers=cpu_count(),
                          sampler=SubsetRandomSampler(list(range(hparams.train_size))),
                         drop_last=True)
    
    def val_dataloader(self):
        return DataLoader(PretrainingDatasetWrapper(stl10_unlabeled,
                                            debug=getattr(self.hparams, "debug", False)),
                          batch_size=self.hparams.batch_size, 
                          shuffle=False,
                          num_workers=cpu_count(),
                          sampler=SequentialSampler(list(range(hparams.train_size + 1, hparams.train_size + hparams.validation_size))),
                         drop_last=True)
    
    def forward(self, X):
        return self.model(X)
    
    def step(self, batch, step_name = "train"):
        (X, Y), y = batch
        embX, projectionX = self.forward(X)
        embY, projectionY = self.forward(Y)
        loss = self.loss(projectionX, projectionY)
        loss_key = f"{step_name}_loss"
        tensorboard_logs = {loss_key: loss}

        return { ("loss" if step_name == "train" else loss_key): loss, 'log': tensorboard_logs,
                        "progress_bar": {loss_key: loss}}
    
    def training_step(self, batch, batch_idx):
        return self.step(batch, "train")
    
    def validation_step(self, batch, batch_idx):
        return self.step(batch, "val")
    
    def validation_end(self, outputs):
        if len(outputs) == 0:
            return {"val_loss": torch.tensor(0)}
        else:
            loss = torch.stack([x["val_loss"] for x in outputs]).mean()
            return {"val_loss": loss, "log": {"val_loss": loss}}

    def configure_optimizers(self):
        optimizer = RMSprop(self.model.parameters(), lr=self.hparams.lr)
        return [optimizer], []

Initial hyper-parameters. Batch size of 128 works fine with EfficientNet-B0 on GTX1070. Note that I’ve limited training dataset to first 10k images from STL10 for convenience of running this blogpost in the form of Jupyter Notebook / Google Colab.

Important! SimCLR greatly benefits from large batch sizes - it should be set to as high value as possible given the GPU / cluster limits.

python

from argparse import Namespace
hparams = Namespace(
    lr=1e-3,
    epochs=50,
    batch_size=160,
    train_size=10000,
    validation_size=1000
)

Finding good initial-learning rate using LRFinder algorithm

I use pytorch-lightning’s built-in LRFinder algorithm to find initial learning rate.

python

module = ImageEmbeddingModule(hparams)
t = pl.Trainer(gpus=1)
lr_finder = t.lr_find(module)
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
CUDA_VISIBLE_DEVICES: [0]

   | Name  | Type            | Params
------------------------------------------
0 | model | ImageEmbedding  | 6 M  
1 | loss  | ContrastiveLoss | 0

python

lr_finder.plot(show=False, suggest=True)
lr_finder.suggestion()
0.000630957344480193

I also use W&B Logging to keep track of my experiments:

python

from pytorch_lightning.loggers import WandbLogger

python

hparams = Namespace(
    lr=0.000630957344480193,
    epochs=10,
    batch_size=160,
    train_size=20000,
    validation_size=1000
)
module = ImageEmbeddingModule(hparams)
logger = WandbLogger(project="simclr-blogpost")
logger.watch(module, log="all", log_freq=50)
trainer = pl.Trainer(gpus=1, logger=logger)

python

trainer.fit(module)
| Name  | Type            | Params
------------------------------------------
0 | model | ImageEmbedding  | 6 M   
1 | loss  | ContrastiveLoss | 0

After the training completes, embeddings are ready for usage for the downstream tasks.

Image classifier on top of SimCLR embeddings

Once the embeddings are trained, they can be used to train the classifier on top of them - either by fine tuning the whole network or by freezing the base network with embeddings and learning linear classifiers on top of it - I show the latter below.

Save the weights of neural network with embeddings

I save the whole network in a form on a checkpoint. Only the internal part of the network will be used later with the classifier (projection layers will be discarded).

python

checkpoint_file = "efficientnet-b0-stl10-embeddings.ckpt"

python

trainer.save_checkpoint(checkpoint_file)

python

trainer.logger.experiment.log_artifact(checkpoint_file, type="model")

Classifier module

Again, I define a custom module - this time it uses already existing embeddings and freezes the base model weights on demand. Note the SimCLRClassifier.embeddings are only the EfficientNet part of the whole network used before - the projection head is discarded.

python

class SimCLRClassifier(nn.Module):
    def __init__(self, n_classes, freeze_base, embeddings_model_path, hidden_size=512):
        super().__init__()
        
        base_model = ImageEmbeddingModule.load_from_checkpoint(embeddings_model_path).model
        
        self.embeddings = base_model.embedding
        
        if freeze_base:
            print("Freezing embeddings")
            for param in self.embeddings.parameters():
                param.requires_grad = False
                
        # Only linear projection on top of the embeddings should be enough
        self.classifier = nn.Linear(in_features=base_model.projection[0].in_features, 
                      out_features=n_classes if n_classes > 2 else 1)

    
    def forward(self, X, *args):
        emb = self.embeddings(X)
        return self.classifier(emb)

Classifier training code

Classifier training code again uses PyTorch lightning, so I’m skipping in-depth explanation.

python

from torch import nn
from torch.optim.lr_scheduler import CosineAnnealingLR

python

class SimCLRClassifierModule(pl.LightningModule):
    def __init__(self, hparams):
        super().__init__()
        hparams = Namespace(**hparams) if isinstance(hparams, dict) else hparams
        self.hparams = hparams
        self.model = SimCLRClassifier(hparams.n_classes, hparams.freeze_base, 
                                      hparams.embeddings_path,
                                      self.hparams.hidden_size)
        self.loss = nn.CrossEntropyLoss()
    
    def total_steps(self):
        return len(self.train_dataloader()) // self.hparams.epochs
    
    def preprocessing(seff):
        return transforms.Compose([
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ])
    
    def get_dataloader(self, split):
        return DataLoader(STL10(".", split=split, transform=self.preprocessing()),
                          batch_size=self.hparams.batch_size, 
                          shuffle=split=="train",
                          num_workers=cpu_count(),
                         drop_last=False)
    
    def train_dataloader(self):
        return self.get_dataloader("train")
    
    def val_dataloader(self):
        return self.get_dataloader("test")
    
    def forward(self, X):
        return self.model(X)
    
    def step(self, batch, step_name = "train"):
        X, y = batch
        y_out = self.forward(X)
        loss = self.loss(y_out, y)
        loss_key = f"{step_name}_loss"
        tensorboard_logs = {loss_key: loss}

        return { ("loss" if step_name == "train" else loss_key): loss, 'log': tensorboard_logs,
                        "progress_bar": {loss_key: loss}}
    
    def training_step(self, batch, batch_idx):
        return self.step(batch, "train")
    
    def validation_step(self, batch, batch_idx):
        return self.step(batch, "val")
    
    def test_step(self, batch, batch_idx):
        return self.step(Batch, "test")
    
    def validation_end(self, outputs):
        if len(outputs) == 0:
            return {"val_loss": torch.tensor(0)}
        else:
            loss = torch.stack([x["val_loss"] for x in outputs]).mean()
            return {"val_loss": loss, "log": {"val_loss": loss}}

    def configure_optimizers(self):
        optimizer = RMSprop(self.model.parameters(), lr=self.hparams.lr)
        schedulers = [
            CosineAnnealingLR(optimizer, self.hparams.epochs)
        ] if self.hparams.epochs > 1 else []
        return [optimizer], schedulers

It’s worth mentioning here that training with a frozen base model gives a great performance boost during training as the gradients need to be calculated only for a single layer. Additionally, by utilizing good embeddings, only a few epochs are required to reach a good quality classifier with single linear projection.

python

hparams_cls = Namespace(
    lr=1e-3,
    epochs=5,
    batch_size=160,
    n_classes=10,
    freeze_base=True,
    embeddings_path="./efficientnet-b0-stl10-embeddings.ckpt",
    hidden_size=512
)
module = SimCLRClassifierModule(hparams_cls)
logger = WandbLogger(project="simclr-blogpost-classifier")
logger.watch(module, log="all", log_freq=10)
trainer = pl.Trainer(gpus=1, max_epochs=hparams_cls.epochs, logger=logger)

python

lr_find_cls = trainer.lr_find(module)
| Name  | Type             | Params
-------------------------------------------
0 | model | SimCLRClassifier | 4 M   
1 | loss  | CrossEntropyLoss | 0  
LR finder stopped early due to diverging loss.

python

lr_find_cls.plot(show=False, suggest=True)
lr_find_cls.suggestion()
0.003981071705534969

python

hparams_cls = Namespace(
    lr=0.003981071705534969,
    epochs=5,
    batch_size=160,
    n_classes=10,
    freeze_base=True,
    embeddings_path="./efficientnet-b0-stl10-embeddings.ckpt",
    hidden_size=512
)
module = SimCLRClassifierModule(hparams_cls)

python

trainer.fit(module)
| Name  | Type             | Params
-------------------------------------------
0 | model | SimCLRClassifier | 4 M   
1 | loss  | CrossEntropyLoss | 0

Evaluation

Here I define a utility function for evaluating the model using the provided data loader. Note the transfer between GPU and CPU and storing all results in-memory might be not effective for large datasets!

python

from sklearn.metrics import classification_report
def evaluate(data_loader, module):
    with torch.no_grad():
        progress = ["/", "-", "\\", "|", "/", "-", "\\", "|"]
        module.eval().cuda()
        true_y, pred_y = [], []
        for i, batch_ in enumerate(data_loader):
            X, y = batch_
            print(progress[i % len(progress)], end="\r")
            y_pred = torch.argmax(module(X.cuda()), dim=1)
            true_y.extend(y.cpu())
            pred_y.extend(y_pred.cpu())
        print(classification_report(true_y, pred_y, digits=3))
        return true_y, pred_y

python

_ = evaluate(module.val_dataloader(), module)

bash

                  precision    recall  f1-score   support
    
               0      0.856     0.864     0.860       800
               1      0.714     0.701     0.707       800
               2      0.903     0.919     0.911       800
               3      0.678     0.599     0.636       800
               4      0.665     0.746     0.703       800
               5      0.633     0.564     0.597       800
               6      0.729     0.781     0.754       800
               7      0.678     0.709     0.693       800
               8      0.868     0.910     0.888       800
               9      0.862     0.801     0.830       800
    
        accuracy                          0.759      8000
       macro avg      0.759     0.759     0.758      8000
    weighted avg      0.759     0.759     0.758      8000

Summary

I hope you find my explanation of the SimCLR framework useful. Feel free to ask anything, comment, play with the notebook and share this post with the ML community!

Additional links and resources

SimCLR paper - arxiv.org/pdf/2002.05709.pdf
EfficientNet paper - arxiv.org/abs/1905.11946
this notebook on GitHub - link
this notebook on Google Colab - link

Training RoBERTa from scratch - the missing guide

Marcin Zabłocki — Sun, 15 Mar 2020 23:00:00 +0000

After hours of research and attempts to understand all of the necessary parts required for one to train custom BERT-like model from scratch using HuggingFace’s Transformers library I came to conclusion that existing blog posts and notebooks are always really vague and do not cover important parts or just skip them like they weren’t there - I will give a few examples, just follow the post.

I’ve decided to get my hands dirty and try to train transformer language model (BERT or other) for Polish language. Here you will find all of the missing pieces that I have encountered while doing it.

Many thanks to Egnyte Inc. for providing me with necessary resources to run the training on powerful machine with 4x NVIDIA P100 GPUs.

TL;DR

This post covers:

what mistakes to avoid
how to prepare own dataset with details - I use Polish Wikipedia and Wolne Lektury datasets
how to prepare training configuration
how to train new tokenizers using HugginFace’s Rust tokenizers
how to use run_language_modeling.py script from HuggingFace library to train your model - what parameters to set, how to deal with the script
how to monitor training process
how to share the model on HuggingFace’s model hub

Getting the dataset

The most important part when dealing with language models is to have solid dataset with text in the language you will be modeling. In my proof of concept I’ve started with Polish Wikipedia and then extended my dataset with Wolne Lektury (repository of public domain polish books and poems).

Preparing Polish Wikipedia dump

So there is a “just get wikipedia” part that noone covers in details but I find it important. First of all, on the wikipedia dumps page there are many files that you can download. I’ve decided to download dump from 2020-02-20 labeled as:

Articles, templates, media/file descriptions, and primary meta-pages, in multiple bz2 streams, 100 pages per stream

which consisted of 7 files with names like: plwiki-20200220-pages-articles-multistream*.xml*.bz2. Why? That kind of split allows to simpify preprocessing - I don’t have to have lots of RAM to processs smaller files. Another benefit is that I can just hold-out one or two of those files for validation phase after my language model learning is complete.

Every file is a huge XML containing articles with mediawiki-specific markup language. I used this tool to extract well formated JSONs from them: https://github.com/attardi/wikiextractor (clone the repo).

So now I have a list of files for both training and evaluation:

bash

    # train.txt
    plwiki-20200220-pages-articles-multistream1.xml-p1p169750
    plwiki-20200220-pages-articles-multistream2.xml-p169751p510662
    plwiki-20200220-pages-articles-multistream3.xml-p510663p1056310
    plwiki-20200220-pages-articles-multistream4.xml-p1056311p1831508
    plwiki-20200220-pages-articles-multistream5.xml-p1831509p3070393
    plwiki-20200220-pages-articles-multistream6.xml-p4570394p4720470

and

bash

    # eval.txt
    plwiki-20200220-pages-articles-multistream6.xml-p3070394p4570393

To extract them just run:

bash

cat train.txt | xargs -I@  python wikiextractor/WikiExtractor.py @ --bytes=100M --json --output="./plwiki-json/train/@"

After extracting you will get the following structure:

bash

├── eval
│   └── plwiki-20200220-pages-articles-multistream6.xml-p3070394p4570393
│       └── AA
│           ├── wiki_00
│           ├── wiki_01
│           ├── wiki_02
│           └── wiki_03
└── train
    ├── plwiki-20200220-pages-articles-multistream1.xml-p1p169750
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       └── wiki_02
    ├── plwiki-20200220-pages-articles-multistream2.xml-p169751p510662
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       └── wiki_02
    ├── plwiki-20200220-pages-articles-multistream3.xml-p510663p1056310
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       └── wiki_02
    ├── plwiki-20200220-pages-articles-multistream4.xml-p1056311p1831508
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       ├── wiki_02
    │       └── wiki_03
    ├── plwiki-20200220-pages-articles-multistream5.xml-p1831509p3070393
    │   └── AA
    │       ├── wiki_00
    │       ├── wiki_01
    │       ├── wiki_02
    │       └── wiki_03
    └── plwiki-20200220-pages-articles-multistream6.xml-p4570394p4720470
        └── AA
            └── wiki_00

Extracting text from wikipedia

After you get JSON lines from wikipedia dump, you need to transform articles in JSON text into plaintext consumable by the language model training script.

Important!

I’ve probably made a mistake here which is important later on! I first tokenized the text by sentences and outputted each sentence in a separate line because of the ambiguity of available resources about dataset preparation. I think that it affects the quality of both tokenization and training. Do not make that mistake.

After digging throughout the HuggingFace’s github I’ve finally got an answer that there should be one document per line. OK - but what if the text is really long? We will be dealing not only with long wiki articles but later on - with books that have thousands of characters and current SOTA language models have limit of around 512-768 tokens (depends on the model). The possible solution for that is to use sliding window over sentences in the input text. For more details see the following thread: https://github.com/huggingface/transformers/issues/2693

To extract the text and tokenize the input I’ve used nltk with custom polish abbreviations - thanks to Krzysztof Sopyła’s awesome gist: https://gist.github.com/ksopyla/f05fe2f48bbc9de895368b8a7863b5c3.

You can find my extraction notebook here: https://gist.github.com/marrrcin/e383b75a5d0dad42048847d97965e037. I decided to skip it here to not pollute the post.

Key takeaways from preprocessing:

tokenize article into sentences
use sliding window over sentences to handle long texts - I used sliding window of size 4

Once the extraction completes, you will have training file like this:

bash

Sumatra Północna Sumatra Północna (indonez. "Sumatera Utara") – prowincja w Indonezji w północnej części Sumatry. Obejmuje również wyspę Nias i wyspy Batu leżące na Oceanie Indyjskim. Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan.
"Sumatera Utara") – prowincja w Indonezji w północnej części Sumatry. Obejmuje również wyspę Nias i wyspy Batu leżące na Oceanie Indyjskim. Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan. Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba.
Obejmuje również wyspę Nias i wyspy Batu leżące na Oceanie Indyjskim. Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan. Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba. Bliżej wybrzeży nizinne tereny w dużym stopniu pokryte bagnami.
Powierzchnia 70 787 km²; 12 391 tys. mieszkańców (2005); stolica Medan. Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba. Bliżej wybrzeży nizinne tereny w dużym stopniu pokryte bagnami. Ludność prowincji tworzą Malajowie wyznający islam, Batakowie będący w większości chrześcijanami (gł. protestantami) oraz przybysze z Jawy, Chin, Indii.
Większą część powierzchni zajmują góry Barisan (Sinabung 2460 m n.p.m.) z licznymi wulkanami i jeziorem Toba. Bliżej wybrzeży nizinne tereny w dużym stopniu pokryte bagnami. Ludność prowincji tworzą Malajowie wyznający islam, Batakowie będący w większości chrześcijanami (gł. protestantami) oraz przybysze z Jawy, Chin, Indii. Główne miasta: Medan, Binjai, Pematang Siantar, Tebing Tinggi

Prepare Wolne Lektury dump

For extraction of polish books I have crawled https://wolnelektury.pl/katalog/ site. It’s pretty straightforward process - all of the books can be downloaded in well formatted TXT files. The preprocessing steps are exactly the same as for wiki, namely: tokenize book into sentences, use sliding window to transform long books into smaller chunks without dropping any data - this time I used sliding window of 8 sentences. Gist of preprocessing notebook: https://gist.github.com/marrrcin/bcc115fbadf79eba9d9c8ca711da9e20.

Training new tokenizer

At this point, I’ve decided to go with RoBERTa model. Model you choose determines the tokenizer that you will have to train. For RoBERTa it’s a ByteLevelBPETokenizer, for BERT it would be BertWordPieceTokenizer (both from tokenizers library). Training the tokenizer is super fast thanks to the Rust implementation that guys at HuggingFace have prepared (great job!).

python


from pathlib import Path

from tokenizers import ByteLevelBPETokenizer
from glob import glob
paths = list(
    glob("your_input_dataset_files/*.txt")
)
# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer(lowercase=False)

# Customize training
tokenizer.train(files=paths, vocab_size=32000, min_frequency=3, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

# Save files to disk
import os
OUT_DIR = "polish_tokenizer_bpe_32k"
os.makedirs(OUT_DIR, exist_ok=True)
tokenizer.save(OUT_DIR, "pl")

Here, the special_tokens part is super important - those tokens need to be exactly in the same order, because otherwise the model will crash!.

For BERT tokenizer however, you will have:

python

from tokenizers import BertWordPieceTokenizer, ByteLevelBPETokenizer
# Initialize a tokenizer
tokenizer = BertWordPieceTokenizer(lowercase=False, handle_chinese_chars=False)

Adjusting the `run_language_modeling.py` script

The version I used and modified comes from v.2.5.1 release of HuggingFace transformers: https://github.com/huggingface/transformers/blob/v2.5.1/examples/run_language_modeling.py

Dataset - check, tokenizer - check. Now it’s time for the model training. Initially, I wanted to use the run_language_modeling.py script from HuggingFace git repo without changes, but it’s it does not support fast Rust-based tokenizers (as for 2020-03-16). Here I mention the minimal changes I made to make it work.

Fast tokenizers support in `run_language_modeling.py`

In order to plug fast tokenizer I’ve trained above into this script I had to modify the LineByLineTextDataset that’s provided there. The final version looks like this:

python

class LineByLineTextDataset(Dataset):
    def __init__(self, t: PreTrainedTokenizer, args, file_path: str, block_size=512):
        assert os.path.isfile(file_path)
        logger.info("Creating features from dataset file at %s", file_path)
        
        # -------------------------- CHANGES START
        bert_tokenizer = os.path.join(args.tokenizer_name, "vocab.txt")
        if os.path.exists(bert_tokenizer):
            logger.info("Loading BERT tokenizer")
            from tokenizers import BertWordPieceTokenizer
            tokenizer = BertWordPieceTokenizer(os.path.join(args.tokenizer_name, "vocab.txt"), handle_chinese_chars=False, lowercase=False)
            tokenizer.enable_truncation(512)
        else:
            from tokenizers import ByteLevelBPETokenizer
            from tokenizers.processors import BertProcessing
            logger.info("Loading RoBERTa tokenizer")
            
            tokenizer = ByteLevelBPETokenizer(
                os.path.join(args.tokenizer_name, "vocab.json"),
                os.path.join(args.tokenizer_name, "merges.txt")
            )
            tokenizer._tokenizer.post_processor = BertProcessing(
                ("</s>", tokenizer.token_to_id("</s>")),
                ("<s>", tokenizer.token_to_id("<s>")),
            )
            tokenizer.enable_truncation(max_length=512)

        logger.info("Reading file %s", file_path)
        with open(file_path, encoding="utf-8") as f:
            lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]

        logger.info("Running tokenization")
        self.examples = tokenizer.encode_batch(lines)
        
        # -------------------------- CHANGES END

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        return torch.tensor(self.examples[i].ids, dtype=torch.long)

So the only changes are in the way of processing input args.tokenizer_name parameter - if the folder has vocab.txt it’s BERT tokenizer, RoBERTa otherwise. Thanks to the tokenizer.encode_batch(lines) the whole dataset is tokenized in parallel.

Important!

If you don’t have enough RAM to fit the whole dataset into memory, you have to implement your own dataset. My input dataset - Polish wikipedia with sliding window of 4 + Wolne Lektury with sliding window of 8 took ~235GB of RAM.

Preparing training configuration

This part is the most tricky one, as there are a lot of magic numbers in the posts / notebooks I’ve found, even in the official one https://huggingface.co/blog/how-to-train. They just skip it… Start with creating the directory for the configuration, say MyRoBERTaConfig. Copy the tokenizer files and rename them. Be sure to finally have:

bash

    config.json
    tokenizer_config.json
    merges.txt
    vocab.json

in this directory.

Training configuration for RoBERTa

You know what? You don’t have to specify any model-specific parameters if you want to go with the defaults! The only thing that you actually need to specify is the vocabulary size, because it determines the output layer size (Language Model is predicting the tokens after all…). For my RoBERTa I have the following config.json:

javascript

{"architectures": ["RobertaForMaskedLM"], "max_position_embeddings": 514, "vocab_size": 32000}

The architectures is self explanatory - this is the language model you will be training, vocab_size comes from the tokenizer you have created.

Important!

The tricky one is max_position_embeddings - no one will tell you that, but if you don’t set this here, the script will just crash. Awesome. I believe that for BERT model it’s not required.

Tokenizer configuration for RoBERTa is simple (tokenizer_config.json). However it’s probably not required, because we’ve overridden the LineByLineTextDataset dataset with tokenizer.

javascript

{"max_len": 512}

Training your own RoBERTa language model

Here I will show two types of run_language_modeling.py script execution - one for training from scratch and the second - for fine tuning.

Training from scratch

This is the usual case for any new language model. First, the script:

bash

#!/bin/bash
export TRAIN_FILE=plwiki.train.txt
export EVAL_FILE=plwiki.eval.txt
# run_language_modeling_with_tokenizers.py -- it's the version with support for fast tokenizers, see above
python run_language_modeling_with_tokenizers.py \
    --train_data_file $TRAIN_FILE \
    --eval_data_file $EVAL_FILE \
    --output_dir ./MyRoBERTa \
    --model_type roberta \
    --mlm \
    --config_name ./MyRoBERTaConfig \
    --tokenizer_name ./MyRoBERTaConfig \
    --do_train \
    --do_eval \
    --line_by_line \
    --learning_rate 1e-5 \
    --num_train_epochs 5 \
    --save_total_limit 20 \
    --save_steps 5000 \
    --per_gpu_train_batch_size 8 \
    --warmup_steps=10000 \
    --logging_steps=100 \
    --gradient_accumulation_steps=4 \
    --seed 666 --block_size=512

Arguments explanation: (only the non-trivial)

--output_dir - here you will get all of the model checkpoints, saved as often as specified in --save_steps argument
--mlm - configure the script to run Masked Language Model task. The script will fortunately yell at you if the --model_type is not appropriate for this task
--line_by_line - we say to use LineByLineTextDataset, the script will use my overridden version with tokenizers
--save_total_limit - how many checkpoint to keep on the disk - the oldest one will be removed once this limit is exceeded. It’s good to set it here in order to not clog your disk
--per_gpu_train_batch_size - setting this parameter might require a few failed runs fro you. If you don’t know how big batch size you can fit into your GPU, just sample your training data, to say 10k lines and launch the script a few times. Monitor the GPU usage by using watch -n 1 nvidia-smi command. Set it to as high number as possible. For Nvidia P100 and 512 --block_size, it’s 8.
--gradient_accumulation_steps - this parameter configures how many steps you will pass in forward pass of the network before running backward pass, it’s useful when you GPU cannot handle large batches
--warmup_steps - configures learning rate scheduler; as per BERT paper recommendation - it’s good to do warmup and then warmdown. Number of steps depends on your training dataset size
--logging_steps - tells how often to store Tensorboard logs - the script only outputs loss and learning rate

Training from checkpoint (finetuning)

After I’ve pretrained my model on wikipedia only, I decided to finetune it further on the Wolne Lektury dataset. Training script has produced a few snapshots, from which every one has the following structure:

bash

├── checkpoint-20000
│   ├── config.json
│   ├── merges.txt
│   ├── optimizer.pt
│   ├── pytorch_model.bin
│   ├── scheduler.pt
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   ├── training_args.bin
│   └── vocab.json

Important!

If you’re changing the input dataset, you cannot use existing scheduler.pt and optimizer.pt as you’re not “resuming” the training. You should remove (but backup first!) those files if you want to fine tune on another dataset.'

OK, now the script itself:

bash

#!/bin/bash
export TRAIN_FILE=wolne-lektury.sliding8.txt
export EVAL_FILE=plwiki.eval.txt

python run_language_modeling_with_tokenizers.py \
    --train_data_file $TRAIN_FILE \
    --eval_data_file $EVAL_FILE \
    --output_dir ./MyRoBERTa_with_another_dataset \
    --model_type roberta \
    --model_name_or_path ./MyRoBERTa/checkpoint-30000 \
    --mlm \
    --config_name ./MyRoBERTaConfig \
    --tokenizer_name ./MyRoBERTaConfig \
    --do_train \
    --do_eval \
    --line_by_line \
    --learning_rate 5e-5 \
    --num_train_epochs 3 \
    --save_total_limit 20 \
    --save_steps 5000 \
    --per_gpu_train_batch_size 8 \
    --warmup_steps=5000 \
    --logging_steps=100 \
    --gradient_accumulation_steps=4 \
    --mlm_probability=0.2 \
    --seed 666 --block_size=512

Arguments - notable mentions:

--mlm_probability=0.2 - this parameter controls the percentage of the tokens you mask during training; default is 0.15, I’ve decided to change it to make the training more difficult to the model
--model_name_or_path ./MyRoBERTa/checkpoint-30000 - this should point to the checkpoint of your previously trained model; remember the note above!

Training phase

I highly recommend you to launch the script using screen tool, which allows to easily spawn and manage multiple SSH sessions - you will be able to disconnect from the server while the training runs. And it will run for days… so brace yourself. You can use screen -UL command (-U for enabling UTF-8 mode for progress bars and -L to log session output to file - it’s helpful when something crashes).

During training, the script will output Tensorboard logs into runs directory. You can monitor them by running tensorboard command:

bash

tensorboard --logdir=runs

It will launch small webserver on localhost:6006 and you will be able to monitor the training.

Training summary for Polish RoBERTa a.k.a PolBERTa

I’ve run my training in three phases:

From scratch on Polish Wikipedia only (1 sentence per line) for 370k steps using learning rate 1e-5 with 10k warmup steps.
Fine tuning on Wolne Lektury only (8 sentences per line) for 60k steps, starting after checkpoint from 1. Learning rate 5e-5 with 5k warmup steps.
Fine tuning on Wolne Lektury (8 sentences per line) and Polish Wikipedia (4 sentences per line) for 30k steps. Learning rate 5-e5 with 10k warmup steps.

Tensorboard for all of the runs can be viewed here https://tensorboard.dev/experiment/aYrAE9uMTxGKRtLUOFjprg/

Exporting the model

If you want to share the model with the NLP community (which I highly encourage you to do!) you need to export it in appropriate format. Prepare the path for your model’s latest checkpoint and then run the following code:

python

from transformers import AutoModelWithLMHead, AutoTokenizer
import os
directory = "/path/to/your/model/checkpoint-30000"
model = AutoModelWithLMHead.from_pretrained(directory)
tokenizer = AutoTokenizer.from_pretrained(directory)
out = "MyBERTa-base-cased-v1"
os.makedirs(out, exist_ok=True)
model.save_pretrained(out)
tokenizer.save_pretrained(out)

Then use transformers-cli util to upload the model:

bash

transformers-cli upload ./MyBERTa-base-cased-v1/

Using my pre-trained model

I’ve uploaded my pre-trained RoBERTa to HuggingFace’s model hub: marrrcin/PolBERTa-base-polish-cased-v1 Usage:

python

from transformers import AutoModelWithLMHead, AutoTokenizer, pipeline

model = AutoModelWithLMHead.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1")
tokenizer = AutoTokenizer.from_pretrained("marrrcin/PolBERTa-base-polish-cased-v1")

fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
fill_mask("Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad <mask>, a postępujesz tak jakbyś miał niecałe dziesięć.")

Output:

javascript

[{'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad sto, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.1013106107711792,
  'token': 1675},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad tysiąc, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.09420681744813919,
  'token': 31144},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad połowę, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.0891065001487732,
  'token': 13627},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad dwadzieścia, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.07204979658126831,
  'token': 18811},
 {'sequence': '<s> Wiadomym jest mi, że masz lat blisko czterdzieści, wyglądasz na blisko trzydzieści, wyobrażasz sobie, że masz nieco ponad godzinę, a postępujesz tak jakbyś miał niecałe dziesięć.</s>',
  'score': 0.059559762477874756,
  'token': 20775}]

Follow up and TODO

Here are a few follow up things and my planned future work on Polish RoBERTa:

prepare model card for HuggingFace's model hub
evaluate PolBERTa on PolEmo2.0 and other downstream tasks
re-train PolBERTa again, using bigger vocab size for tokenizer
fine tune model further on another open datasets

If you find any bug in the post or something is not clear - feel free to post a comment, I will be glad to improve this short practical guide.

Thanks to…

I would like to thank Egnyte Inc. for providing me with required resources to train the language model and also Darek Kłeczek (creator of original PolBERT) for tips related to training BERT.

Additional links and resources

PolBERT - original BERT-base-uncased LM trained by Darek Kłeczek
HuggingFaces original guide for training new language models

Use Pandas DataFrames in scikit-learn FeatureUnion and Pipelines

Marcin Zabłocki — Fri, 27 Apr 2018 22:00:00 +0000

Pandas is a popular data wrangling library among data engineers and data scientists who use Python. When it comes to machine learning with Python, scikit-learn is the top pick for writing not only Jupyter-based experiments but also for full machine learning pipelines. Unfortunately, Pandas and scikit-learn does not play well together when you try to use scikit-learn’s Pipeline and FeatureUnion abstractions. This post is here to help you deal with those problems using just a few lines of code.

TL;DR

How to use pandas DataFrame with scikit-learn FeatureUnion and Pipelines.

Environment

Python 3.6.4 (should work on 2.7.* too!)
scikit-learn 0.19.1
pandas 0.22.0

The problem

Problem with using DataFrames with scikit-learn starts to emerge when you want to preserve abilities that pandas provide i.e column names, ease of indexing, mapping and filtering. By default, scikti-learn does suport using DataFrames, however it strips them down to plain numpy arrays, which lack of programmers favourite DataFrame features.

Consider the following code:

python

raw_data = load_iris()
data = pd.DataFrame(raw_data["data"], columns=raw_data["feature_names"])

pipeline = FeatureUnion([
    ("1", make_pipeline(
        FunctionTransformer(lambda X: X.loc[:, ["sepal length (cm)"]]),
        # other transformations
    )),
    ("2", make_pipeline(
        FunctionTransformer(lambda X: X.loc[:, ["sepal width (cm)"]]),
        # other transformations
    ))
])

X = pipeline.fit_transform(data)
print(X["sepal length (cm)"].mean())
print(X["sepal width (cm)"].mean())

The idea is to apply different transformations to different columns and then merge them together for further processing / machine learning process. Looks readable and seems like it should work but if you try to run it you will get the following error:

python

FunctionTransformer(...),
AttributeError: 'numpy.ndarray' object has no attribute 'loc'

The reason for that is FunctionTransformer strips any non-numpy array objects to numpy arrays and we lose DataFrame objects there.

Solution part 1

Solution for transforming columns is simple, we need to have custom Transformer class which preserves object types:

python

class PandasTransform(TransformerMixin, BaseEstimator):
    def __init__(self, fn):
        self.fn = fn

    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None, copy=None):
        return self.fn(X)

You need to change previous FunctionTransformer to PandasTransformer:

python

pipeline = FeatureUnion([
    ("1", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal length (cm)"]]),
        # other transformations
    )),
    ("2", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal width (cm)"]]),
        # other transformations
    ))
])

X = pipeline.fit_transform(data)
print(X["sepal length (cm)"].mean())
print(X["sepal width (cm)"].mean())

It’s better this time, because the code executes the transform and variable X contains transformed data. The problem is that different exception occured:

python

IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

Why? Because scikit-learn FeatureUnion did the same stripping to DataFrames like the FunctionTransform did previously.

Solution part 2: Pandas DataFrame Featue Union

Scikit-learn team is aware of this missing feature, however GitHub issue is still unresolved. Solution for FeatureUnion problem is just to add the support of Pandas DataFrames to it.

python

class PandasFeatureUnion(FeatureUnion):
    def fit_transform(self, X, y=None, **fit_params):
        self._validate_transformers()
        result = Parallel(n_jobs=self.n_jobs)(
            delayed(_fit_transform_one)(trans, weight, X, y,
                                        **fit_params)
            for name, trans, weight in self._iter())

        if not result:
            # All transformers are None
            return np.zeros((X.shape[0], 0))
        Xs, transformers = zip(*result)
        self._update_transformer_list(transformers)
        if any(sparse.issparse(f) for f in Xs):
            Xs = sparse.hstack(Xs).tocsr()
        else:
            Xs = self.merge_dataframes_by_column(Xs)
        return Xs

    def merge_dataframes_by_column(self, Xs):
        return pd.concat(Xs, axis="columns", copy=False)

    def transform(self, X):
        Xs = Parallel(n_jobs=self.n_jobs)(
            delayed(_transform_one)(trans, weight, X)
            for name, trans, weight in self._iter())
        if not Xs:
            # All transformers are None
            return np.zeros((X.shape[0], 0))
        if any(sparse.issparse(f) for f in Xs):
            Xs = sparse.hstack(Xs).tocsr()
        else:
            Xs = self.merge_dataframes_by_column(Xs)
        return Xs

The above code seems to be long, but there are only a few things goin on there:

PandasFeatureUnion class extends scikit-learn built-in FeatureUnion
It overrides transform and fit_transform functions

Both transform and fit_transform functions were copied from scikit-learn source and the only changes were

from:

python

if any(sparse.issparse(f) for f in Xs):
    Xs = sparse.hstack(Xs).tocsr()
else:
    Xs = np.hstack(Xs)

to:

python

if any(sparse.issparse(f) for f in Xs):
    Xs = sparse.hstack(Xs).tocsr()
else:
    Xs = self.merge_dataframes_by_column(Xs)

Results

After combining PandasTransform and PandasFeatureUnion, pipeline should work like charm and output data will be still in DataFrame format.

python

pipeline = PandasFeatureUnion([
    ("1", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal length (cm)"]]),
        # other transformations
    )),
    ("2", make_pipeline(
        PandasTransform(lambda X: X.loc[:, ["sepal width (cm)"]]),
        # other transformations
    ))
])

X = pipeline.fit_transform(data)
print(X["sepal length (cm)"].mean())
print(X["sepal width (cm)"].mean())

Summary

With just a few tricks and hacks you can use DataFrames alongside scikit-learn Pipelines to build your machine learning pipelines. I hope this post will help you!

Additional links & resources

full project on my GitHub: https://github.com/marrrcin/pandas-feature-union
scikit-learn FeatureUnion tutorial: http://scikit-learn.org/stable/auto_examples/hetero_feature_union.html

Schedule Python Apache Beam DataFlow pipeline using App Engine CRON service

Marcin Zabłocki — Fri, 30 Jun 2017 22:00:00 +0000

Apache Beam is a relatively new framework, which claims to deliver unified, parallel processing model for the data. Apache Beam with Google DataFlow can be used in various data processing scenarios like: ETLs (Extract Transform Load), data migrations and machine learning pipelines. This post explains how to run Apache Beam Python pipeline using Google DataFlow and then how to deploy this pipeline to App Engine in order to run it i.e on a daily basis using App Engine CRON service. There are some tutorials explaining how to create basic pipelines and how to run them using Apache Beam DataFlow Runner, but there is not much information how to deploy those pipelines to be runned without user interaction, which should be the first thing to do, since no one wants to trigger big data pipelines by hand.

As an example pipeline I will show wordcount on a few books of Shakespeare.

TL;DR

Use App Engine CRON to launch Google DataFlow pipeline written in Apache Beam Python SDK. Link to github: https://github.com/marrrcin/python-beam-dataflow-cron

Prerequisites

Python 2.7.12+
Apache Beam SDK for Python 2.0.0
Google Cloud Platform account to run the pipeline in DataFlow and store results in Cloud Storage
Google Cloud SDK for Python
basic knowledge of Apache Beam

The pipeline

The idea behind wordcount pipeline in Apache Beam is simple - actually it’s almost the same as for MapReduce, but simpler in the implementation, because of the Apache Beam capabilities and expressiveness:

read all input files
tokenize the text
for every token, emit tuple (word, 1) - mapping phase
group tuples by word
sum values for every group - aggregation phase
format and save output

Pipeline implementation

Here is python code for the pipeline described above. Comments and code explanation is below.

python

from __future__ import print_function, absolute_import
import apache_beam as beam
from apache_beam.io import ReadFromText
from apache_beam.io import WriteToText
import logging
logging.basicConfig(level=logging.INFO)


class FindWords(beam.DoFn):
    def process(self, element):
        import re as regex
        return regex.findall(r"[A-Za-z\']+", element)


class CountWordsTransform(beam.PTransform):
    def expand(self, p_collection):
        # p_collection is collection of loaded lines
        return (p_collection
                | "Find words" >> (beam.ParDo(FindWords()).with_input_types(unicode))
                | "Pair With One" >> beam.Map(lambda word: (word, 1))
                | "Group By" >> beam.GroupByKey()
                | "Aggregate Groups" >> beam.Map(lambda (word, ones): (word, sum(ones))))


def run():
    import time
    gcs_path = "gs://marcin-playground/dataflow"
    pipeline =  beam.Pipeline(runner="DataflowRunner", argv=[
        "--project", "your-project-name",
        "--staging_location", ("%s/staging_location" % gcs_path),
        "--temp_location", ("%s/temp" % gcs_path),
        "--output", ("%s/output" % gcs_path),
        "--setup_file", "./setup.py"
    ])
    (pipeline
     | "Load" >> ReadFromText("gs://marcin-playground/books/*.txt")
     | "Count Words" >> CountWordsTransform()
     | "Format Output" >> beam.Map(lambda (word, count): "{0}: {1}".format(word, count))
     | "Save" >> WriteToText("{0}/output/wordcount{1}".format(gcs_path, int(time.time())))
     )
    pipeline.run()

Besides the required imports, first important part is the run() method - it’s an entry point to the pipeline. Whan you can observe there is that it does not follow the Apache Beam documentation convention, where beam.Pipeline objects are wrapped inside the python with clause. I did it on purpose - thanks to that, the code will not wait for the pipeline to complete, but rather it will dispatch the job to Google DataFlow and return (see “DataflowRunner” in the constructor). Other configuration parameters are self-descriptive and they are required by the Beam SDK.

First, the pipeline reads all book files (link to those files is at the end of this post) from Google Cloud Storage - you need to create GCS bucket and put the input files into the folders in order to use them in the pipeline.

Count Words is a second step. It uses my implementation of beam.PTransform class, which can be used to join multiple transforms into single high-level piece. The implementation itself consists of 4 steps:

Finding the words - for simplicity, I assume that word only contains lower and upper case letters and/or an apostrophe. In the FindWords class, which is a custom implementation of beam.DoFn, the def process(self, element) function is responsible of transforming each line (element parameter) of the readed books in to something else, in this case - the output will be a list of words.
Next, each list of words is transformed into tuples in form (word, 1), it’s just a matter of invoking beam.Map built-in function
Third step is grouping tuples by word, so we can then aggregate (count) them in total
Last step is to acctualy aggregate them, thanks to the fact that we’ve emmited tuples with 1, we can just sum them to obtain the total number of occurences for each word

Format Output does only one simple thing - transforms each tuple in form (word, number of occurences) to readable string in format: my word: 123.

Last step is to Save results into file. For the case of running the pipeline multiple times, output folder will have current timestamp in the name.

Deploying Python DataFlow pipeline to Google App Engine

Now when the pipeline is ready, we can prepare python web application which will trigger DataFlow jobs. For this task, I will use Flask framework.

Project structure

Start with creating the project structure:

Description of the structure:

app.yaml - definition of Google App Engine application
cron.yaml - definition of Google App Engine CRON job
main.py - entry point for the web application
dataflow_pipeline module - the pipeline described above, wrapped inside the python module
requirements.txt - file for pip package manager, which contains list of all required packages to run the application and the pipeline
appengine_config.py - standard App Engine python configuration file
lib - local folder with all pip-installed packages from requirements.txt file
setup.py - setup file, which is required for DataFlow to be able to run the code from the pipeline - when you use libraries like pandas in your pipeline code, you must notify the DataFlow to install this package on every worker machine that will be running the pipeline - that’s why the setup.py file is required

Configuration

What is really important in the requirements.txt file. For my pipeline, it’s contents are:

python

Flask==0.12.2
apache-beam[gcp]==2.0.0
gunicorn==19.7.1
google-cloud-dataflow==2.0.0

In order to make the App Engine application to work, all of the required packages are needed to be deployed with the application, that’s why you need to install them locally by invoking:

powershell

pip install -r requirements.txt -t lib

Also note that apache-beam package is used with [gcp] feature, which will install the capabilities of running the pipeline in DataFlow.

Configuration file for the application looks like this: app.yaml

html

runtime: python
env: flex

entrypoint: gunicorn -b :$PORT main:app

runtime_config:
  python_version: 2

The file describes that we want to use App Engine Flex environment, with python 2.7 runtime.

IMPORTANT: Apache Beam pipelines will not work on App Engine Standard environment, due to some filestystem I/O operations. You will most likely encounter this exception there:

powershell

IOError: [Errno 30] Read-only file system: '/dev/null'
at <module> (/base/data/home/apps/project/20170626t141416.402244965699518249/lib/dill/dill.py:167)
at <module> (/base/data/home/apps/project/20170626t141416.402244965699518249/lib/dill/__init__.py:27)

The solution is just to use Flex environment.

Flask application for starting DataFlow pipeline

Here is the implementation of Flask application with single endpoint: /start-dataflow.

python

from __future__ import print_function, absolute_import
from flask import Flask, request
import logging
import dataflow_pipeline.wordcountpipeline as pipeline

app = Flask(__name__)


@app.route("/start-dataflow")
def start_dataflow():
    is_cron = request.headers.get('X-Appengine-Cron', False)
    if not is_cron:
        return 'Bad Request', 400

    try:
        pipeline.run()
        return "Pipeline started", 200
    except Exception as e:
        logging.exception(e)
        return "Error: <pre>{}</pre>".format(e), 500


@app.errorhandler(500)
def server_error(e):
    logging.exception('An error occurred during a request.')
    return """
    An internal error occurred: <pre>{}</pre>
    See logs for full stacktrace.
    """.format(e), 500


if __name__ == '__main__':
    app.run(host='127.0.0.1', port=8080, debug=True)

From the code above, the most important part is the def start_dataflow() function. It checks for the presence of X-Appengine-Cron HTTP request header. Thanks to that, the app can know, whether the request came from App Engine CRON service or just from external call. For the debugging purposes, you can comment out this part, but remember to NOT DEPLOY application without this check, because then, any user will be able to trigger your pipeline and potentially increase your billing a lot!

After asserting the presence of HTTP header, Apache Beam pipeline is spawned by calling run() method which was defined in our pipeline above. If everything goes fine, web application will return status code 200. If something will go wrong, you will get status code 500 with stacktrace, which you can use to hunt down the issue.

Defining App Engine CRON service to trigger python DataFlow pipeline

Last thing to do is to define cron.yaml file with CRON job description:

html

cron:
- description: trigger dataflow test
  url: /start-dataflow
  schedule: every 30 mins
  target: default

It’s a simple file, which will send GET request to /start-dataflow endpoint every 30 minutes. Of course you can modify the schedule as needed.

Deployment

Deploy web application by invoking:

powershell

gcloud app deploy app.yaml

Note that for large pipelines with lots of external libraries, deploying App Engine Flex environment application might take a while.

Deploy CRON service by invoking:

powershell

gcloud app deploy cron.yaml

Go to Google Cloud Platform Console (https://console.cloud.google.com) and open App Engine tab. You should see your app there.
Under App Engine tab, open Task queues > Cron Jobs - the CRON service should be visible there. You can trigger it manually by using [Run now] button.

Executing the pipeline

After successful job dispatch, you can open GCP Console and navigate to DataFlow tab to see all of your finished / running jobs:

For each job, you can view it’s exectuion statistics, logs and data flow visualization:

Summary

That’s it! I hope this end-to-end example will help you deploy your jobs to Google DataFlow. Please share or leave comment if you like it!

Pro tips

when you want to use library in Apache Beam pipeline i.e in your beam.DoFn implementation, always use local imports (in the method body)
if you use custom python libraries (installed through pip) you must create setup.py file with all the packages listed there, because Google DataFlow workers are deployed only with limited number of preinstalled packages

Additional links & resources

full project on my github: https://github.com/marrrcin/python-beam-dataflow-cron
Shakespeare’s book collection: http://www.gutenberg.org/ebooks/author/65
Apache Beam Python Guide: https://beam.apache.org/documentation/programming-guide/

Sentiment analysis of tweets with Python, NLTK, word2vec & scikit-learn

Marcin Zabłocki — Thu, 25 May 2017 22:00:00 +0000

This post describes full machine learning pipeline used for sentiment analysis of twitter posts divided by 3 categories: positive, negative and neutral. For this task I used python with: scikit-learn, nltk, pandas, word2vec and xgboost packages.

TL;DR

Detailed description & report of tweets sentiment analysis using machine learning techniques in Python. With details, but this is not a tutorial.

Jupyter Notebook of this post

This post is compiled version of Jupyter Notebook, which you can download here: https://github.com/marrrcin/ml-twitter-sentiment-analysis

Abstract

The goal of this project was to predict sentiment for the given Twitter post using Python. Sentiment analysis can predict many different emotions attached to the text, but in this report only 3 major were considered: positive, negative and neutral. The training dataset was small (just over 5900 examples) and the data within it was highly skewed, which greatly impacted on the difficulty of building good classifier. After creating a lot of custom features, utilizing both bag-of-words and word2vec representations and applying the Extreme Gradient Boosting algorithm, the classification accuracy at level of 58% was achieved.

Used Python Libraries

Data was pre-processed using pandas, gensim and numpy libraries and the learning/validating process was built with scikit-learn. Plots were created using plotly.

python

from collections import Counter
import nltk
import pandas as pd
from emoticons import EmoticonDetector
import re as regex
import numpy as np
import plotly
from plotly import graph_objs
from sklearn.metrics import f1_score, precision_score, recall_score, accuracy_score
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV, RandomizedSearchCV
from time import time
import gensim

# plotly configuration
plotly.offline.init_notebook_mode()

Notebook code convention

This report was first prepared as a classical Python project using object oriented programming with maintainability in mind. In order to show this project as a Jupyter Notebook, the classes had to be splitted into multiple code-cells. In order to do so, the classes are suffixed with _PurposeOfThisSnippet name and they inherit one from another. The final class will be then run and the results will be shown.

Data source

The input data consisted two CSV files: train.csv (5971 tweets) and test.csv (4000 tweets) - one for training and one for testing. Format of the data was the following (test data didn’t contain Category column):

Id	Category	Tweet
635930169241374720	neutral	IOS 9 App Transport Security. Mm need to check if my 3rd party network pod supports it

All tweets are in english, so it simplifies the processing and analysis.

Data preprocessing

Loading the data

python

class TwitterData_Initialize():
    data = []
    processed_data = []
    wordlist = []

    data_model = None
    data_labels = None
    is_testing = False
    
    def initialize(self, csv_file, is_testing_set=False, from_cached=None):
        if from_cached is not None:
            self.data_model = pd.read_csv(from_cached)
            return

        self.is_testing = is_testing_set

        if not is_testing_set:
            self.data = pd.read_csv(csv_file, header=0, names=["id", "emotion", "text"])
            self.data = self.data[self.data["emotion"].isin(["positive", "negative", "neutral"])]
        else:
            self.data = pd.read_csv(csv_file, header=0, names=["id", "text"],dtype={"id":"int64","text":"str"},nrows=4000)
            not_null_text = 1 ^ pd.isnull(self.data["text"])
            not_null_id = 1 ^ pd.isnull(self.data["id"])
            self.data = self.data.loc[not_null_id & not_null_text, :]

        self.processed_data = self.data
        self.wordlist = []
        self.data_model = None
        self.data_labels = None

The code snippet above is prepared, to load the data form the given file for further processing, or just read already preprocessed file from the cache. There’s also a distinction between processing testing and training data. As the test.csv file was full of empty entries, they were removed. Additional class properties such as data_model, wordlist etc. will be used further.

python

data = TwitterData_Initialize()
data.initialize("data\\train.csv")
data.processed_data.head(5)

	id	emotion	text
0	635769805279248384	negative	Not Available
1	635930169241374720	neutral	IOS 9 App Transport Security. Mm need to check...
2	635950258682523648	neutral	Mar if you have an iOS device, you should down...
3	636030803433009153	negative	@jimmie_vanagon my phone does not run on lates...
4	636100906224848896	positive	Not sure how to start your publication on iOS?...

Data distribution

First thing that can be done as soon as the data is loaded is to see the data distribution. The training set had the following distribution:

python

df = data.processed_data
neg = len(df[df["emotion"] == "negative"])
pos = len(df[df["emotion"] == "positive"])
neu = len(df[df["emotion"] == "neutral"])
dist = [
    graph_objs.Bar(
        x=["negative","neutral","positive"],
        y=[neg, neu, pos],
)]
plotly.offline.iplot({"data":dist, "layout":graph_objs.Layout(title="Sentiment type distribution in training set")})

Preprocessing steps

The targed of the following preprocessing is to create a Bag-of-Words representation of the data. The steps will execute as follows:

Cleansing
1. Remove URLs
2. Remove usernames (mentions)
3. Remove tweets with Not Available text
4. Remove special characters
5. Remove numbers
Text processing
1. Tokenize
2. Transform to lowercase
3. Stem
Build word list for Bag-of-Words

Cleansing

For the purpose of cleansing, the TwitterCleanup class was created. It consists methods allowing to execute all of the tasks show in the list above. Most of those is done using regular expressions. The class exposes it’s interface through iterate() method - it yields every cleanup method in proper order.

python

class TwitterCleanuper:
    def iterate(self):
        for cleanup_method in [self.remove_urls,
                               self.remove_usernames,
                               self.remove_na,
                               self.remove_special_chars,
                               self.remove_numbers]:
            yield cleanup_method

    @staticmethod
    def remove_by_regex(tweets, regexp):
        tweets.loc[:, "text"].replace(regexp, "", inplace=True)
        return tweets

    def remove_urls(self, tweets):
        return TwitterCleanuper.remove_by_regex(tweets, regex.compile(r"http.?://[^\s]+[\s]?"))

    def remove_na(self, tweets):
        return tweets[tweets["text"] != "Not Available"]

    def remove_special_chars(self, tweets):  # it unrolls the hashtags to normal words
        for remove in map(lambda r: regex.compile(regex.escape(r)), [",", ":", "\"", "=", "&", ";", "%", "$",
                                                                     "@", "%", "^", "*", "(", ")", "{", "}",
                                                                     "[", "]", "|", "/", "\\", ">", "<", "-",
                                                                     "!", "?", ".", "'",
                                                                     "--", "---", "#"]):
            tweets.loc[:, "text"].replace(remove, "", inplace=True)
        return tweets

    def remove_usernames(self, tweets):
        return TwitterCleanuper.remove_by_regex(tweets, regex.compile(r"@[^\s]+[\s]?"))

    def remove_numbers(self, tweets):
        return TwitterCleanuper.remove_by_regex(tweets, regex.compile(r"\s?[0-9]+\.?[0-9]*"))

The loaded tweets can be now cleaned.

python

class TwitterData_Cleansing(TwitterData_Initialize):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        
    def cleanup(self, cleanuper):
        t = self.processed_data
        for cleanup_method in cleanuper.iterate():
            if not self.is_testing:
                t = cleanup_method(t)
            else:
                if cleanup_method.__name__ != "remove_na":
                    t = cleanup_method(t)

        self.processed_data = t

python

data = TwitterData_Cleansing(data)
data.cleanup(TwitterCleanuper())
data.processed_data.head(5)

	id	emotion	text
1	635930169241374720	neutral	IOS App Transport Security Mm need to check if...
2	635950258682523648	neutral	Mar if you have an iOS device you should downl...
3	636030803433009153	negative	my phone does not run on latest IOS which may ...
4	636100906224848896	positive	Not sure how to start your publication on iOS ...
5	636176272947744772	neutral	Two Dollar Tuesday is here with Forklift Quick...

Tokenization & stemming

For the text processing, nltk library is used. First, the tweets are tokenized using nlkt.word_tokenize and then, stemming is done using PorterStemmer as the tweets are 100% in english.

python

class TwitterData_TokenStem(TwitterData_Cleansing):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        
    def stem(self, stemmer=nltk.PorterStemmer()):
        def stem_and_join(row):
            row["text"] = list(map(lambda str: stemmer.stem(str.lower()), row["text"]))
            return row

        self.processed_data = self.processed_data.apply(stem_and_join, axis=1)

    def tokenize(self, tokenizer=nltk.word_tokenize):
        def tokenize_row(row):
            row["text"] = tokenizer(row["text"])
            row["tokenized_text"] = [] + row["text"]
            return row

        self.processed_data = self.processed_data.apply(tokenize_row, axis=1)

python

data = TwitterData_TokenStem(data)
data.tokenize()
data.stem()
data.processed_data.head(5)

	id	emotion	text	tokenized_text
1	635930169241374720	neutral	[io, app, transport, secur, mm, need, to, chec...	[IOS, App, Transport, Security, Mm, need, to, ...
2	635950258682523648	neutral	[mar, if, you, have, an, io, devic, you, shoul...	[Mar, if, you, have, an, iOS, device, you, sho...
3	636030803433009153	negative	[my, phone, doe, not, run, on, latest, io, whi...	[my, phone, does, not, run, on, latest, IOS, w...
4	636100906224848896	positive	[not, sure, how, to, start, your, public, on, ...	[Not, sure, how, to, start, your, publication,...
5	636176272947744772	neutral	[two, dollar, tuesday, is, here, with, forklif...	[Two, Dollar, Tuesday, is, here, with, Forklif...

Building the wordlist

The wordlist (dictionary) is build by simple count of occurences of every unique word across all of the training dataset.

Before building the final wordlist for the model, let’s take a look at the non-filtered version:

python

words = Counter()
for idx in data.processed_data.index:
    words.update(data.processed_data.loc[idx, "text"])

words.most_common(5)

# output:
# [('the', 3744), ('to', 2477), ('i', 1667), ('a', 1620), ('on', 1557)]

The most commont words (as expected) are the typical english stopwords. We will filter them out, however, as purpose of this analysis is to determine sentiment, words like “not” and “n’t” can influence it greatly. Having this in mind, this word will be whitelisted.

python

stopwords=nltk.corpus.stopwords.words("english")
whitelist = ["n't", "not"]
for idx, stop_word in enumerate(stopwords):
    if stop_word not in whitelist:
        del words[stop_word]
words.most_common(5)

# output
# [('may', 1027), ('tomorrow', 764), ('day', 526), ('go', 499), ('thi', 495)]

Still, there are some words that seem too be occuring to many times, let’s filter them. After some analysis, the lower bound was set to 3.

The wordlist is also saved to the csv file, so the same words can be used for the testing set.

python

class TwitterData_Wordlist(TwitterData_TokenStem):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        
    whitelist = ["n't","not"]
    wordlist = []
        
    def build_wordlist(self, min_occurrences=3, max_occurences=500, stopwords=nltk.corpus.stopwords.words("english"),
                       whitelist=None):
        self.wordlist = []
        whitelist = self.whitelist if whitelist is None else whitelist
        import os
        if os.path.isfile("data\\wordlist.csv"):
            word_df = pd.read_csv("data\\wordlist.csv")
            word_df = word_df[word_df["occurrences"] > min_occurrences]
            self.wordlist = list(word_df.loc[:, "word"])
            return

        words = Counter()
        for idx in self.processed_data.index:
            words.update(self.processed_data.loc[idx, "text"])

        for idx, stop_word in enumerate(stopwords):
            if stop_word not in whitelist:
                del words[stop_word]

        word_df = pd.DataFrame(data={"word": [k for k, v in words.most_common() if min_occurrences < v < max_occurences],
                                     "occurrences": [v for k, v in words.most_common() if min_occurrences < v < max_occurences]},
                               columns=["word", "occurrences"])

        word_df.to_csv("data\\wordlist.csv", index_label="idx")
        self.wordlist = [k for k, v in words.most_common() if min_occurrences < v < max_occurences]

python

data = TwitterData_Wordlist(data)
data.build_wordlist()

python

words = pd.read_csv("data\\wordlist.csv")
x_words = list(words.loc[0:10,"word"])
x_words.reverse()
y_occ = list(words.loc[0:10,"occurrences"])
y_occ.reverse()

dist = [
    graph_objs.Bar(
        x=y_occ,
        y=x_words,
        orientation="h"
)]
plotly.offline.iplot({"data":dist, "layout":graph_objs.Layout(title="Top words in built wordlist")})

Bag-of-words

The data is ready to transform it to bag-of-words representation.

python

class TwitterData_BagOfWords(TwitterData_Wordlist):
    def __init__(self, previous):
        self.processed_data = previous.processed_data
        self.wordlist = previous.wordlist
    
    def build_data_model(self):
        label_column = []
        if not self.is_testing:
            label_column = ["label"]

        columns = label_column + list(
            map(lambda w: w + "_bow",self.wordlist))
        labels = []
        rows = []
        for idx in self.processed_data.index:
            current_row = []

            if not self.is_testing:
                # add label
                current_label = self.processed_data.loc[idx, "emotion"]
                labels.append(current_label)
                current_row.append(current_label)

            # add bag-of-words
            tokens = set(self.processed_data.loc[idx, "text"])
            for _, word in enumerate(self.wordlist):
                current_row.append(1 if word in tokens else 0)

            rows.append(current_row)

        self.data_model = pd.DataFrame(rows, columns=columns)
        self.data_labels = pd.Series(labels)
        return self.data_model, self.data_labels

Let’s take a look at the data and see, which words are the most common for particular sentiments.

python

data = TwitterData_BagOfWords(data)
bow, labels = data.build_data_model()
bow.head(5)

	label	wa_bow	not_bow	...
0	neutral	0	0	...
1	neutral	0	0	...
2	negative	1	1	...
3	positive	0	1	...
4	neutral	0	0	...

5 rows × 2185 columns

python

grouped = bow.groupby(["label"]).sum()
words_to_visualize = []
sentiments = ["positive","negative","neutral"]
#get the most 7 common words for every sentiment
for sentiment in sentiments:
    words = grouped.loc[sentiment,:]
    words.sort_values(inplace=True,ascending=False)
    for w in words.index[:7]:
        if w not in words_to_visualize:
            words_to_visualize.append(w)
            
            
#visualize it
plot_data = []
for sentiment in sentiments:
    plot_data.append(graph_objs.Bar(
            x = [w.split("_")[0] for w in words_to_visualize],
            y = [grouped.loc[sentiment,w] for w in words_to_visualize],
            name = sentiment
    ))
    
plotly.offline.iplot({
        "data":plot_data,
        "layout":graph_objs.Layout(title="Most common words across sentiments")
    })

Some of the most common words show high distinction between classes like go and see and other are occuring in similiar amount for every class (plan, obama).

None of the most common words is unique to the negative class. At this point, it’s clear that skewed data distribution will be a problem in distinguishing negative tweets from the others.

Classification

First of all, lets establish seed for random numbers generators.

python

import random
seed = 666
random.seed(seed)

The following utility function will train the classifier and show the F1, precision, recall and accuracy scores.

python

def test_classifier(X_train, y_train, X_test, y_test, classifier):
    log("")
    log("===============================================")
    classifier_name = str(type(classifier).__name__)
    log("Testing " + classifier_name)
    now = time()
    list_of_labels = sorted(list(set(y_train)))
    model = classifier.fit(X_train, y_train)
    log("Learing time {0}s".format(time() - now))
    now = time()
    predictions = model.predict(X_test)
    log("Predicting time {0}s".format(time() - now))

    precision = precision_score(y_test, predictions, average=None, pos_label=None, labels=list_of_labels)
    recall = recall_score(y_test, predictions, average=None, pos_label=None, labels=list_of_labels)
    accuracy = accuracy_score(y_test, predictions)
    f1 = f1_score(y_test, predictions, average=None, pos_label=None, labels=list_of_labels)
    log("=================== Results ===================")
    log("            Negative     Neutral     Positive")
    log("F1       " + str(f1))
    log("Precision" + str(precision))
    log("Recall   " + str(recall))
    log("Accuracy " + str(accuracy))
    log("===============================================")

    return precision, recall, accuracy, f1

def log(x):
    #can be used to write to log file
    print(x)

Experiment 1: BOW + Naive Bayes

It is nice to see what kind of results we might get from such simple model. The bag-of-words representation is binary, so Naive Bayes Classifier seems like a nice algorithm to start the experiments.

The experiment will be based on 7:3 train:test stratified split.

python

from sklearn.naive_bayes import BernoulliNB
X_train, X_test, y_train, y_test = train_test_split(bow.iloc[:, 1:], bow.iloc[:, 0],
                                                    train_size=0.7, stratify=bow.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, BernoulliNB())

markdown

    ===============================================
    Testing BernoulliNB
    Learing time 0.451092004776001s
    Predicting time 0.1190040111541748s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.38949672  0.45072993  0.71369782]
    Precision[ 0.45408163  0.48431373  0.65906623]
    Recall   [ 0.34099617  0.42150171  0.77820513]
    Accuracy 0.579594345421
    ===============================================

Result with accuracy at level of 58% seems to be quite nice result for such basic algorithm like Naive Bayes (having in mind that random classifier would yield result of around 33% accuracy). This performance may not hold for the final testing set. In order to see how the NaiveBayes performs in more general cases, 8-fold crossvalidation is used. The 8 fold is used, to optimize speed of testing on my 8-core machine.

python

def cv(classifier, X_train, y_train):
    log("===============================================")
    classifier_name = str(type(classifier).__name__)
    now = time()
    log("Crossvalidating " + classifier_name + "...")
    accuracy = [cross_val_score(classifier, X_train, y_train, cv=8, n_jobs=-1)]
    log("Crosvalidation completed in {0}s".format(time() - now))
    log("Accuracy: " + str(accuracy[0]))
    log("Average accuracy: " + str(np.array(accuracy[0]).mean()))
    log("===============================================")
    return accuracy

python

nb_acc = cv(BernoulliNB(), bow.iloc[:,1:], bow.iloc[:,0])

markdown

    ===============================================
    Crossvalidating BernoulliNB...
    Crosvalidation completed in 4.4375975131988525s
    Accuracy: [ 0.54639175  0.48820059  0.28023599  0.31415929  0.32743363  0.50073855
      0.47119645  0.53106509]
    Average accuracy: 0.432427668406
    ===============================================

This result no longer looks optimistic. For some of the splits, Naive Bayes classifier showed performance below the performance of random classifier.

Additional features

In order to not push any other aglorithm to the limit on the current data model, let’s try to add some features that might help to classify tweets.

A common sense suggest that special characters like exclamation marks and the casing might be important in the task of determining the sentiment. The following features will be added to the data model:

Feature name	Explanation
Number of uppercase	people tend to express with either positive or negative emotions by using A LOT OF UPPERCASE WORDS
Number of !	exclamation marks are likely to increase the strength of opinion
Number of ?	might distinguish neutral tweets - seeking for information
Number of positive emoticons	positive emoji will most likely not occur in the negative tweets
Number of negative emoticons	inverse to the one above
Number of …	commonly used in commenting something
Number of quotations	same as above
Number of mentions	sometimes people put a lot of mentions on positive tweets, to share something good
Number of hashtags	just for the experiment
Number of urls	similiar to the number of mentions

Extraction of those features must be done before any preprocessing happens.

For the purpose of emoticons, the EmoticonDetector class is created. The file emoticons.txt contains list of positive and negative emoticons, which are used.

python

class EmoticonDetector:
    emoticons = {}

    def __init__(self, emoticon_file="data\\emoticons.txt"):
        from pathlib import Path
        content = Path(emoticon_file).read_text()
        positive = True
        for line in content.split("\n"):
            if "positive" in line.lower():
                positive = True
                continue
            elif "negative" in line.lower():
                positive = False
                continue

            self.emoticons[line] = positive

    def is_positive(self, emoticon):
        if emoticon in self.emoticons:
            return self.emoticons[emoticon]
        return False

    def is_emoticon(self, to_check):
        return to_check in self.emoticons

python

class TwitterData_ExtraFeatures(TwitterData_Wordlist):
    def __init__(self):
        pass
    
    def build_data_model(self):
        extra_columns = [col for col in self.processed_data.columns if col.startswith("number_of")]
        label_column = []
        if not self.is_testing:
            label_column = ["label"]

        columns = label_column + extra_columns + list(
            map(lambda w: w + "_bow",self.wordlist))
        
        labels = []
        rows = []
        for idx in self.processed_data.index:
            current_row = []

            if not self.is_testing:
                # add label
                current_label = self.processed_data.loc[idx, "emotion"]
                labels.append(current_label)
                current_row.append(current_label)

            for _, col in enumerate(extra_columns):
                current_row.append(self.processed_data.loc[idx, col])

            # add bag-of-words
            tokens = set(self.processed_data.loc[idx, "text"])
            for _, word in enumerate(self.wordlist):
                current_row.append(1 if word in tokens else 0)

            rows.append(current_row)

        self.data_model = pd.DataFrame(rows, columns=columns)
        self.data_labels = pd.Series(labels)
        return self.data_model, self.data_labels
    
    def build_features(self):
        def count_by_lambda(expression, word_array):
            return len(list(filter(expression, word_array)))

        def count_occurences(character, word_array):
            counter = 0
            for j, word in enumerate(word_array):
                for char in word:
                    if char == character:
                        counter += 1

            return counter

        def count_by_regex(regex, plain_text):
            return len(regex.findall(plain_text))

        self.add_column("splitted_text", map(lambda txt: txt.split(" "), self.processed_data["text"]))

        # number of uppercase words
        uppercase = list(map(lambda txt: count_by_lambda(lambda word: word == word.upper(), txt),
                             self.processed_data["splitted_text"]))
        self.add_column("number_of_uppercase", uppercase)

        # number of !
        exclamations = list(map(lambda txt: count_occurences("!", txt),
                                self.processed_data["splitted_text"]))

        self.add_column("number_of_exclamation", exclamations)

        # number of ?
        questions = list(map(lambda txt: count_occurences("?", txt),
                             self.processed_data["splitted_text"]))

        self.add_column("number_of_question", questions)

        # number of ...
        ellipsis = list(map(lambda txt: count_by_regex(regex.compile(r"\.\s?\.\s?\."), txt),
                            self.processed_data["text"]))

        self.add_column("number_of_ellipsis", ellipsis)

        # number of hashtags
        hashtags = list(map(lambda txt: count_occurences("#", txt),
                            self.processed_data["splitted_text"]))

        self.add_column("number_of_hashtags", hashtags)

        # number of mentions
        mentions = list(map(lambda txt: count_occurences("@", txt),
                            self.processed_data["splitted_text"]))

        self.add_column("number_of_mentions", mentions)

        # number of quotes
        quotes = list(map(lambda plain_text: int(count_occurences("'", [plain_text.strip("'").strip('"')]) / 2 +
                                                 count_occurences('"', [plain_text.strip("'").strip('"')]) / 2),
                          self.processed_data["text"]))

        self.add_column("number_of_quotes", quotes)

        # number of urls
        urls = list(map(lambda txt: count_by_regex(regex.compile(r"http.?://[^\s]+[\s]?"), txt),
                        self.processed_data["text"]))

        self.add_column("number_of_urls", urls)

        # number of positive emoticons
        ed = EmoticonDetector()
        positive_emo = list(
            map(lambda txt: count_by_lambda(lambda word: ed.is_emoticon(word) and ed.is_positive(word), txt),
                self.processed_data["splitted_text"]))

        self.add_column("number_of_positive_emo", positive_emo)

        # number of negative emoticons
        negative_emo = list(map(
            lambda txt: count_by_lambda(lambda word: ed.is_emoticon(word) and not ed.is_positive(word), txt),
            self.processed_data["splitted_text"]))

        self.add_column("number_of_negative_emo", negative_emo)
        
    def add_column(self, column_name, column_content):
        self.processed_data.loc[:, column_name] = pd.Series(column_content, index=self.processed_data.index)

python

data = TwitterData_ExtraFeatures()
data.initialize("data\\train.csv")
data.build_features()
data.cleanup(TwitterCleanuper())
data.tokenize()
data.stem()
data.build_wordlist()
data_model, labels = data.build_data_model()

Logic behind extra features

Let’s see how (some) of the extra features separate the data set. Some of them, i.e number exclamation marks, number of pos/neg emoticons do this really well. Despite of the good separation, those features sometimes occur only on small subset of the training dataset.

python

sentiments = ["positive","negative","neutral"]
plots_data_ef = []
for what in map(lambda o: "number_of_"+o,["positive_emo","negative_emo","exclamation","hashtags","question"]):
    ef_grouped = data_model[data_model[what]>=1].groupby(["label"]).count()
    plots_data_ef.append({"data":[graph_objs.Bar(
            x = sentiments,
            y = [ef_grouped.loc[s,:][0] for s in sentiments],
    )], "title":"How feature \""+what+"\" separates the tweets"})
    

for plot_data_ef in plots_data_ef:
    plotly.offline.iplot({
            "data":plot_data_ef["data"],
            "layout":graph_objs.Layout(title=plot_data_ef["title"])
    })

Experiment 2: extended features + Random Forest

As a second attempt on the classification the Random Forest will be used.

python

from sklearn.ensemble import RandomForestClassifier
X_train, X_test, y_train, y_test = train_test_split(data_model.iloc[:, 1:], data_model.iloc[:, 0],
                                                    train_size=0.7, stratify=data_model.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, RandomForestClassifier(random_state=seed,n_estimators=403,n_jobs=-1))

markdown

    ===============================================
    Testing RandomForestClassifier
    Learing time 6.9144287109375s
    Predicting time 0.21802711486816406s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.24501425  0.47944007  0.70340909]
    Precision[ 0.47777778  0.49192101  0.63163265]
    Recall   [ 0.16475096  0.46757679  0.79358974]
    Accuracy 0.575291948371
    ===============================================

The accuracy for the initial split was lower than the one for the Naive Bayes, but let’s see what happens during crossvalidation:

python

rf_acc = cv(RandomForestClassifier(n_estimators=403,n_jobs=-1, random_state=seed),data_model.iloc[:, 1:], data_model.iloc[:, 0])

markdown

    ===============================================
    Crossvalidating RandomForestClassifier...
    Crosvalidation completed in 70.09595036506653s
    Accuracy: [ 0.54344624  0.50442478  0.37905605  0.27876106  0.37905605  0.52141802
      0.5155096   0.55621302]
    Average accuracy: 0.459735602399
    ===============================================

It looks better, however it’s still not much above accuracy of the random classifier and barely better than Naive Bayes classifier.

We can observe a low recall level of the RandomForest classifier for the negative class, which may be caused by the data skewness.

More features - word2vec

The overall performance of the previous classifiers could be enhanced by performing time-consuming parameters adjustments, however there’s not guarantee on how big the gain will be.

If the out-of-the-shelf methods did not performed well, it seems that there’s not much in the data itself. The next idea to add more into data model is to use word2vec representation of a tweet to perform classification.

The word2vec allows to transform words into vectors of numbers. Those vectors represent abstract features, that describe the word similarities and relationships (i.e co-occurence).

What is the best in the word2vec is that operations on the vectors approximately keep the characteristics of the words, so that joining (averaging) vectors from the words from sentence procude vector that is likely to represent the general topic of the sentence.

A lot of pre-trained word2vec models exists, and some of them were trained on huge volumes of data. For the purpose of this analysis, the one trained on over 2 billion of tweets with 200 dimensions (one vector consists of 200 numbers) is used. The pre-trained model can be downloaded here: https://github.com/3Top/word2vec-api

From GloVe to word2vec

In order to use GloVe-trained model in gensim library, it needs to be converted to word2vec format. The only difference between those formats is that word2vec text files starts with two numbers: number of lines in file and number of dimensions. The file glove.twitter.27B.200d.txt does not contain those lines.

Unfortunaltely, this text file size is over 1.9GB and text editors cannot be used to open and modify it in reasonable amount of time, this C# snippet adds this required line (sorry that it’s not Python, but I was having memory problems with encoding of the file in Python. It’s required to use x64 target):

csharp

using (var fileStream = new FileStream("glove.twitter.27B.200d.txt", FileMode.Open,FileAccess.ReadWrite))
{
    var lines = new LinkedList<string>();
    using (var streamReader = new StreamReader(fileStream))
    {
        while (!streamReader.EndOfStream)
        {
            lines.AddLast(streamReader.ReadLine());
        }
    }
    lines.AddFirst("1193514 200");
    File.WriteAllLines("word2vec.twitter.27B.200d.txt.txt", lines);
}

### Already modified GloVe for file The file that has the first line appended can be downloaded from here (622MB 7-zip file, ultra compression): https://marcin.egnyte.com/dl/KiIlEYaFiU/word2vec.twitter.27B.200d.7z_

Using Word2Vec

The following class exposes a easy to use interface over the word2vec API from gensim library:

python

class Word2VecProvider(object):
    word2vec = None
    dimensions = 0

    def load(self, path_to_word2vec):
        self.word2vec = gensim.models.Word2Vec.load_word2vec_format(path_to_word2vec, binary=False)
        self.word2vec.init_sims(replace=True)
        self.dimensions = self.word2vec.vector_size

    def get_vector(self, word):
        if word not in self.word2vec.vocab:
            return None

        return self.word2vec.syn0norm[self.word2vec.vocab[word].index]

    def get_similarity(self, word1, word2):
        if word1 not in self.word2vec.vocab or word2 not in self.word2vec.vocab:
            return None

        return self.word2vec.similarity(word1, word2)

python

word2vec = Word2VecProvider()

# REPLACE PATH TO THE FILE
word2vec.load("C:\\__\\machinelearning\\glove.twitter.27B.200d.txt")

Extra features from word2vec

Besides the 200 additional features from the word2vec representation, I had an idea of 3 more features. If word2vec allows to find similarity between words, that means it can find similarity to the specific emotion-representing words. The first idea was to compute similarity of the whole tweet with words from labels: positive, negative, neutral. Since the purpose was to find the sentiment, I thought that it will be better to find similarity with more expressive words such as: good and bad. For the neutral sentiment, I’ve used word information, since most of the tweets with neutral sentiment were giving the information.

The features were builded by computing mean similarity of the whole tweet to the given word. Then, those mean values were normalized to [0;1] in order to deal with different word count across tweets.

Final data model

The final data model will contain:

extra text features (number of: !, ?, :-) etc)
word2vec similarity to “good”, “bad” and “information” words
word2vec 200 dimension averaged representation of a tweet
bag-of-word representation of a tweet

python

class TwitterData(TwitterData_ExtraFeatures):
    
    def build_final_model(self, word2vec_provider, stopwords=nltk.corpus.stopwords.words("english")):
        whitelist = self.whitelist
        stopwords = list(filter(lambda sw: sw not in whitelist, stopwords))
        extra_columns = [col for col in self.processed_data.columns if col.startswith("number_of")]
        similarity_columns = ["bad_similarity", "good_similarity", "information_similarity"]
        label_column = []
        if not self.is_testing:
            label_column = ["label"]

        columns = label_column + ["original_id"] + extra_columns + similarity_columns + list(
            map(lambda i: "word2vec_{0}".format(i), range(0, word2vec_provider.dimensions))) + list(
            map(lambda w: w + "_bow",self.wordlist))
        labels = []
        rows = []
        for idx in self.processed_data.index:
            current_row = []

            if not self.is_testing:
                # add label
                current_label = self.processed_data.loc[idx, "emotion"]
                labels.append(current_label)
                current_row.append(current_label)

            current_row.append(self.processed_data.loc[idx, "id"])

            for _, col in enumerate(extra_columns):
                current_row.append(self.processed_data.loc[idx, col])

            # average similarities with words
            tokens = self.processed_data.loc[idx, "tokenized_text"]
            for main_word in map(lambda w: w.split("_")[0], similarity_columns):
                current_similarities = [abs(sim) for sim in
                                        map(lambda word: word2vec_provider.get_similarity(main_word, word.lower()), tokens) if
                                        sim is not None]
                if len(current_similarities) <= 1:
                    current_row.append(0 if len(current_similarities) == 0 else current_similarities[0])
                    continue
                max_sim = max(current_similarities)
                min_sim = min(current_similarities)
                current_similarities = [((sim - min_sim) / (max_sim - min_sim)) for sim in
                                        current_similarities]  # normalize to <0;1>
                current_row.append(np.array(current_similarities).mean())

            # add word2vec vector
            tokens = self.processed_data.loc[idx, "tokenized_text"]
            current_word2vec = []
            for _, word in enumerate(tokens):
                vec = word2vec_provider.get_vector(word.lower())
                if vec is not None:
                    current_word2vec.append(vec)

            averaged_word2vec = list(np.array(current_word2vec).mean(axis=0))
            current_row += averaged_word2vec

            # add bag-of-words
            tokens = set(self.processed_data.loc[idx, "text"])
            for _, word in enumerate(self.wordlist):
                current_row.append(1 if word in tokens else 0)

            rows.append(current_row)

        self.data_model = pd.DataFrame(rows, columns=columns)
        self.data_labels = pd.Series(labels)
        return self.data_model, self.data_labels

python

td = TwitterData()
td.initialize("data\\train.csv")
td.build_features()
td.cleanup(TwitterCleanuper())
td.tokenize()
td.stem()
td.build_wordlist()
td.build_final_model(word2vec)

td.data_model.head(5)

	label	original_id	number_of_uppercase	number_of_question	number_of_mentions	number_of_urls	...
0	neutral	635930169241374720	2	0	0	1	...
1	neutral	635950258682523648	0	0	0	1	...
2	negative	636030803433009153	2	0	1	0	...
3	positive	636100906224848896	0	1	0	1	...
4	neutral	636176272947744772	4	0	0	1	...

5 rows × 2399 columns

python

data_model = td.data_model
data_model.drop("original_id",axis=1,inplace=True)

Logic behind word2vec extra features

In order to show, why the 3 extra features built using word2vec might help in sentiment analysis. The chart below shows how many tweets from given category were dominating (had highest value) on similarity to those words. Although the words doesn’t seem to separate the sentiments themselves, the differences between them in addition to other parameters, may help the classification process - i.e when tweet has highest value on good_similarity it’s more likely for it to be classified to have positive sentiment.

python

columns_to_plot = ["bad_similarity", "good_similarity", "information_similarity"]
bad, good, info = columns_to_plot
sentiments = ["positive","negative","neutral"]
only_positive = data_model[data_model[good]>=data_model[bad]]
only_positive = only_positive[only_positive[good]>=only_positive[info]].groupby(["label"]).count()

only_negative = data_model[data_model[bad] >= data_model[good]]
only_negative = only_negative[only_negative[bad] >= only_negative[info]].groupby(["label"]).count()

only_info = data_model[data_model[info]>=data_model[good]]
only_info = only_info[only_info[info]>=only_info[bad]].groupby(["label"]).count()

plot_data_w2v = []
for sentiment in sentiments:
    plot_data_w2v.append(graph_objs.Bar(
            x = ["good","bad", "information"],
            y = [only_positive.loc[sentiment,:][0], only_negative.loc[sentiment,:][0], only_info.loc[sentiment,:][0]],
            name = "Number of dominating " + sentiment
    ))
    
plotly.offline.iplot({
        "data":plot_data_w2v,
        "layout":graph_objs.Layout(title="Number of tweets dominating on similarity to: good, bad, information")
    })

Experiment 3: full model + Random Forest

The model is now complete. With a lot of new features, the learning algorithms should perform totally differently on the new data set.

python

X_train, X_test, y_train, y_test = train_test_split(data_model.iloc[:, 1:], data_model.iloc[:, 0],
                                                    train_size=0.7, stratify=data_model.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, RandomForestClassifier(n_estimators=403,n_jobs=-1, random_state=seed))
rf_acc = cv(RandomForestClassifier(n_estimators=403,n_jobs=-1,random_state=seed),data_model.iloc[:, 1:], data_model.iloc[:, 0])

markdown

    ===============================================
    Testing RandomForestClassifier
    Learing time 3.579137086868286s
    Predicting time 0.2350320816040039s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.08695652  0.48138056  0.72029835]
    Precision[ 0.8         0.51456311  0.61622607]
    Recall   [ 0.04597701  0.45221843  0.86666667]
    Accuracy 0.585740626921
    ===============================================
    ===============================================
    Crossvalidating RandomForestClassifier...
    Crosvalidation completed in 41.57810711860657s
    Accuracy: [ 0.58468336  0.52654867  0.54129794  0.54867257  0.51622419  0.5465288
      0.58493353  0.56508876]
    Average accuracy: 0.551747226492
    ===============================================

As we can see, the average accuracy from crossvalidation is almost 58%, and the results from the crossvalidation runs are more stable - they never drop below 51%. It might seem that the algorithm will perform pretty well on the testing set. There’s a one problem with that - recall for the 7:3 split shows, that only about 3% of the negative tweets from the whole set were recognized properly - it seems like the algorithm is good with recognition of postive vs. neutral cases, but it does really poor job when it comes to recognize negative sentiment. Let’s try something else.

Experiment 4: full model + XGBoost

XGBoost is relatively new machine learning algorithm based on decision trees and boosting. It is highly scalable and provides results, which are often higher than those obtained using popular algorithms such as Random Forest or SVM.

Important: XGBoost exposes scikit-learn interface, but it needs to be installed as an additional python package. See this page to see more: https://xgboost.readthedocs.io/en/latest/build.html

python

from xgboost import XGBClassifier as XGBoostClassifier

python


X_train, X_test, y_train, y_test = train_test_split(data_model.iloc[:, 1:], data_model.iloc[:, 0],
                                                    train_size=0.7, stratify=data_model.iloc[:, 0],
                                                    random_state=seed)
precision, recall, accuracy, f1 = test_classifier(X_train, y_train, X_test, y_test, XGBoostClassifier(seed=seed))

markdown

    ===============================================
    Testing XGBClassifier
    Learing time 94.12349128723145s
    Predicting time 0.2379899024963379s
    =================== Results ===================
                Negative     Neutral     Positive
    F1       [ 0.33595801  0.50823938  0.72674419]
    Precision[ 0.53333333  0.51675485  0.66489362]
    Recall   [ 0.24521073  0.5         0.80128205]
    Accuracy 0.60356484327
    ===============================================

60% accuracy is the highest result yet. Still, it’s only a result on the splitted training set, let’s see what happens when we crossvalidate this algorithm (with default parameters).

python

xgb_acc = cv(XGBoostClassifier(seed=seed),data_model.iloc[:, 1:], data_model.iloc[:, 0])

markdown

    ===============================================
    Crossvalidating XGBClassifier...
    Crosvalidation completed in 335.4184989929199s
    Accuracy: [ 0.62592047  0.53834808  0.50737463  0.47492625  0.42182891  0.53471196
      0.56277696  0.54585799]
    Average accuracy: 0.526468157158
    ===============================================

Averaged accuracy seems to be lower and less stable than for the Random Forest, but given over 8 times better recall for the negative cases, the XGBoost seems to be a good starting point for the final classifier.

Finding best parameters for XGBoost

Warning the code bellow executes for a really long time. On computer with i7@3.4GHz processor it took over 90 minutes to perform 5 iterations of RandomizedSearchCV, so depending on your resources, you can find even better parameters.

The following utility functions are used to find the best parameters for the XGBoost using RandomizedSearchCV funciton.

python

def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidates = np.flatnonzero(results['rank_test_score'] == i)
        for candidate in candidates:
            log("Model with rank: {0}".format(i))
            log("Mean validation score: {0:.3f} (std: {1:.3f})".format(
                results['mean_test_score'][candidate],
                results['std_test_score'][candidate]))
            log("Parameters: {0}".format(results['params'][candidate]))
            log("")

def best_fit(X_train, y_train, n_iter=5):
    
    parameters = {
        "n_estimators":[103,201, 403],
        "max_depth":[3,10,15, 30],
        "objective":["multi:softmax","binary:logistic"],
        "learning_rate":[0.05, 0.1, 0.15, 0.3]
    }

    rand_search = RandomizedSearchCV(XGBoostClassifier(seed=seed),param_distributions=parameters,
                                     n_iter=n_iter,scoring="accuracy",
                                     n_jobs=-1,cv=8)

    import time as ttt
    now = time()
    log(ttt.ctime())
    rand_search.fit(X_train, y_train)
    report(rand_search.cv_results_, 10)
    log(ttt.ctime())
    log("Search took: " + str(time() - now))

python

best_fit(data_model.iloc[:, 1:], data_model.iloc[:, 0], n_iter=10)

For 10 runs and 12400s running time, the following parameters were discovered:

markdown

Model with rank: 1
Mean validation score: 0.550 (std: 0.034)
Parameters: {'n_estimators': 403, 'max_depth': 10, 'objective': 'binary:logistic', 'learning_rate': 0.15}

Model with rank: 2
Mean validation score: 0.546 (std: 0.044)
Parameters: {'learning_rate': 0.05, 'n_estimators': 201, 'objective': 'multi:softmax', 'max_depth': 10}

Model with rank: 2
Mean validation score: 0.546 (std: 0.035)
Parameters: {'n_estimators': 403, 'max_depth': 15, 'objective': 'binary:logistic', 'learning_rate': 0.1}

Model with rank: 4
Mean validation score: 0.545 (std: 0.033)
Parameters: {'n_estimators': 403, 'max_depth': 30, 'objective': 'binary:logistic', 'learning_rate': 0.15}

Model with rank: 5
Mean validation score: 0.543 (std: 0.033)
Parameters: {'max_depth': 30, 'n_estimators': 103, 'objective': 'multi:softmax', 'learning_rate': 0.3}

Model with rank: 6
Mean validation score: 0.542 (std: 0.041)
Parameters: {'n_estimators': 103, 'max_depth': 30, 'objective': 'binary:logistic', 'learning_rate': 0.1}

Model with rank: 7
Mean validation score: 0.542 (std: 0.046)
Parameters: {'n_estimators': 103, 'max_depth': 10, 'objective': 'binary:logistic', 'learning_rate': 0.05}

Model with rank: 8
Mean validation score: 0.541 (std: 0.033)
Parameters: {'learning_rate': 0.1, 'n_estimators': 103, 'objective': 'binary:logistic', 'max_depth': 10}

Model with rank: 9
Mean validation score: 0.509 (std: 0.068)
Parameters: {'n_estimators': 403, 'max_depth': 3, 'objective': 'multi:softmax', 'learning_rate': 0.1}

Model with rank: 10
Mean validation score: 0.504 (std: 0.068)
Parameters: {'max_depth': 3, 'n_estimators': 403, 'objective': 'multi:softmax', 'learning_rate': 0.15}

Test data classification

After finding best cross-validated parameter for the XGBoost, it’s time to load the test data and predict sentiment for them. Final classifier will be trained on the whole training set. End score will be revealed when the Angry Tweets competition will end (https://inclass.kaggle.com/c/angry-tweets).

The data will be exported to CSV file in format containing two columns: Id, Category. There are 4000 test samples with unknown distribution of the sentiment labels.

python

test_data = TwitterData()
test_data.initialize("data\\test.csv", is_testing_set=True)
test_data.build_features()
test_data.cleanup(TwitterCleanuper())
test_data.tokenize()
test_data.stem()
test_data.build_wordlist()
test_data.build_final_model(word2vec)

test_data.data_model.head(5)

	original_id	number_of_uppercase	number_of_exclamation	number_of_question	number_of_hashtags	number_of_mentions	...
0	628949369883000832	0	0	1	0	1	...
1	628976607420645377	1	1	0	0	1	...
2	629023169169518592	0	0	0	0	0	...
3	629179223232479232	0	0	0	0	0	...
4	629186282179153920	1	0	1	2	2	...

5 rows × 2398 columns

python

test_model = test_data.data_model

data_model = td.data_model

xgboost = XGBoostClassifier(seed=seed,n_estimators=403,max_depth=10,objective="binary:logistic",learning_rate=0.15)
xgboost.fit(data_model.iloc[:,1:],data_model.iloc[:,0])
predictions = xgboost.predict(test_model.iloc[:,1:])

python

test_model.head(5)

	original_id	number_of_uppercase	number_of_exclamation	number_of_question	number_of_hashtags	number_of_mentions	...
0	628949369883000832	0	0	1	0	1	...
1	628976607420645377	1	1	0	0	1	...
2	629023169169518592	0	0	0	0	0	...
3	629179223232479232	0	0	0	0	0	...
4	629186282179153920	1	0	1	2	2	...

5 rows × 2398 columns

python

results = pd.DataFrame([],columns=["Id","Category"])
results["Id"] = test_model["original_id"].astype("int64")
results["Category"] = predictions
results.to_csv("results_xgb.csv",index=False)

Feature importance

Let’s take a look at the final model feature importance.

Output list is really long, so the print line is commented in the code block

python

features = {}
for idx, fi in enumerate(xgboost.feature_importances_):
    features[test_model.columns[1+idx]] = fi

important = []
for f in sorted(features,key=features.get,reverse=True):
    important.append((f,features[f]))
    # print(f + " " + str(features[f]))

It’s not surprise that the word2vec and related good/bad/information similarity features were the most important, because the classification performance was greatly improved after switching to this representation.

What is interesting is that a lot of custom-crafted features (number_of_*) were also highly important, beating a lot of features which came from bag-of-words representation. Here’s the chart for the easiness of reading:

python

to_show = list(filter(lambda f: not f[0].startswith("word2vec") and not f[0].endswith("_similarity"),important))[:20]
to_show.reverse()
features_importance = [
    graph_objs.Bar(
        x=[f[1] for f in to_show],
        y=[f[0] for f in to_show],
        orientation="h"
)]
plotly.offline.iplot({"data":features_importance, "layout":graph_objs.Layout(title="Most important features in the final model",
      margin=graph_objs.Margin(
           l=200,
        pad=3
    ),)})

XGBoost considered custom features as important, so it also confirmed that some non-word features of a text can also be used to predict the sentiment. Most of them were even more important than the actual presence of some emotion-expressing words in the text.

Summary

Experiment showed that prediction of text sentiment is a non-trivial task for machine learning. A lot of preprocessing is required just to be able to run any algorithm and see - usually not great - results. Main problem for sentiment analysis is to craft the machine representation of the text. Simple bag-of-words was definitely not enough to obtain satisfying results, thus a lot of additional features were created basing on common sense (number of emoticons, exclamation marks etc.). Word2vec representation significantly raised the predictions quality. I think that a slight improvement in classification accuracy for the given training dataset could be developed, but since it contained highly skewed data (small number of negative cases), the difference will be probably in the order of a few percent. The thing that could possibly improve classification results will be to add a lot of additional examples (increase training dataset), because given 5971 examples obviously do not contain every combination of words usage, moreover - a lot of emotion-expressing words surely are missing.

machine-learning on Marcin Zabłocki blog

Twitter Sentiment Analysis on BigQuery using ONNX + Huggingface Transformers

TL;DR

What is not in the scope of this blogpost

What you will find below

Prerequisites

Dataset

Preparing the model

How to fix it? - take 1

How to fix it? - take 2

Creating BigQuery ONNX model

Huggingface tokenizers in BigQuery

Huggingface tokenizer as BigQuery Remote Function

DistilBERT inference on BigQuery

Scalling the inference

ONNX models vs Remote Functions

Summary

Additional links & resources

Deploy MLflow models on BigQuery

TL;DR

⚠️ What is not in the scope of this blogpost

What you will find below

Prerequisites

Training a simple model with TPOT’s AutoML

Creating BigQuery Remote Function with Fast API

BigQuery Remote Function contract

Remote Function endpoint

Testing out the endpoint locally

Deploying Remote Function to Cloud Run

Building a Docker container

Create Cloud Run service

Connecting from BigQuery to Cloud Run

(option 1) From UI

(option 2) From CLI

Running the inference from BigQuery

💡Pro-tips

MLflow model saving

Multiple Remote Functions in a single endpoint

Summary

Additional links & resources

Understanding & implementing SimCLR in PyTorch - an ELI5 guide

TL;DR

Understanding SimCLR framework

Contrastive loss function

Theory behind contrastive loss function

Contrastive loss function - implementation in PyTorch, ELI5 version

Explanation

Contrastive loss function - implementation in PyTorch, vectorized version

Pre-training image embeddings using SimCLR with EfficientNet

Utility functions for image augmentations

Automatic data augmentation wrapper

SimCLR neural network for embeddings

Finding good initial-learning rate using LRFinder algorithm

Image classifier on top of SimCLR embeddings

Save the weights of neural network with embeddings

Classifier module

Classifier training code

Evaluation

Summary

Additional links and resources

Training RoBERTa from scratch - the missing guide

TL;DR

Getting the dataset

Preparing Polish Wikipedia dump

Extracting text from wikipedia

Prepare Wolne Lektury dump

Training new tokenizer

Adjusting the run_language_modeling.py script

Fast tokenizers support in run_language_modeling.py

Preparing training configuration

Training configuration for RoBERTa

Training your own RoBERTa language model

Training from scratch

Training from checkpoint (finetuning)

Training phase

Training summary for Polish RoBERTa a.k.a PolBERTa

Exporting the model

Using my pre-trained model

Follow up and TODO

Thanks to…

Adjusting the `run_language_modeling.py` script

Fast tokenizers support in `run_language_modeling.py`