Keep in mind that for bigger batch sizes you have to pad to either longest text's lenght or to model's max supported length (which usually for BERT-like models is `512`).
The second dimension contains internal representation of each of the input tokens, so for properly tokenized data, i.e:
```python
t = "Mówiąc otwarcie jesteście reliktami."
enc = tokenizer.encode_plus(t)
token_representations = base_model(torch.tensor(enc["input_ids"]).unsqueeze(0))[0][0]
print(enc["input_ids"])
print(tokenizer.decode(enc["input_ids"]))
print(f"Length: {len(enc['input_ids'])}")
print(token_representations.shape)
```
[0, 355, 304, 1748, 71, 12029, 28398, 807, 2578, 19022, 18, 2]
| text | label | |
|---|---|---|
| 0 | Na samym wejściu hotel śmierdzi . W pokojach j... | meta_minus_m |
| 1 | Ze swoimi dziećmi ( 10 lat i 2 latka ) chodzę ... | meta_plus_m |
| 2 | Chciał em zestaw do podstawowych prac coś przy... | meta_minus_m |
| 3 | Drogie Mamy . Mam o tym laktatorze takie same ... | meta_minus_m |
| 4 | własciwie nic się nie dowidział em , podczas b... | meta_minus_m |
From the plot above we can guess that something between 1e-5 and 1e-4 would be a good learning rate, as everyhing higher results in increased loss.
To ensure the right spot was chosen, you can plot it:
```python
lr = 4e-5
lr
```
4e-05
```python
lr_finder.plot(show_lr=lr)
```
## Training the sentiment analysis classifier on PolEmo2.0
We know the learning rate, we have the code for the model. Now it's time to run it.
```python
hparams = Namespace(
train_path=train_path,
val_path=val_path,
test_path=test_path,
batch_size=8,
warmup_steps=100,
epochs=3,
lr=lr,
accumulate_grad_batches=1
)
module = TrainingModule(hparams)
```
```python
# just to get rid of any leftovers before the training - useful when you run different cells multiple times
import gc; gc.collect()
torch.cuda.empty_cache()
```
### Using Trainer from PyTorch Lightning
The `pl.Trainer` provides all-in-one solution for training, validation and testing. There is a lot of useful parameters for almost every aspect of training you can think of, everything is well documented here.
```python
trainer = pl.Trainer(gpus=1, max_epochs=hparams.epochs, progress_bar_refresh_rate=10,
accumulate_grad_batches=hparams.accumulate_grad_batches)
```
```python
trainer.fit(module)
```
## Evaluating the trained model
As this dataset is small, I've used scikit-learn to get the overall classification report from the model.
Important!
> Be aware that for big datasets this approach might not be optimal as it transfers the data from GPU to CPU for the whole dataset and puts every prediction in memory. For big datasets it's better to implement metrics inside of the `pl.LightningModule` when the dataset is processed in batches.
> Ensure that you disable gradients and put model into evaluation state (`module.eval()`) to disable dropouts during prediction.
```python
with torch.no_grad():
progress = ["/", "-", "\\", "|", "/", "-", "\\", "|"]
module.eval()
true_y, pred_y = [], []
for i, batch_ in enumerate(module.test_dataloader()):
(X, attn), y = batch_
batch = (X.cuda(), attn.cuda())
print(progress[i % len(progress)], end="\r")
y_pred = torch.argmax(module(batch), dim=1)
true_y.extend(y.cpu())
pred_y.extend(y_pred.cpu())
print("\n" + "_" * 80)
print(classification_report(true_y, pred_y, target_names=label2int.keys(), digits=6))
```
```python
________________________________________________________________________________
precision recall f1-score support
meta_minus_m 0.880878 0.828909 0.854103 339
meta_plus_m 0.867521 0.894273 0.880694 227
meta_zero 0.957983 0.966102 0.962025 118
meta_amb 0.574324 0.625000 0.598592 136
accuracy 0.832927 820
macro avg 0.820177 0.828571 0.823854 820
weighted avg 0.837433 0.832927 0.834617 820
```
## Additional links and resources
* this notebook in Google Colab - https://colab.research.google.com/drive/1sajgpLTrTJDzRSlxycy8aE6ysxGqaT18
* this notebook on GitHub - https://github.com/marrrcin/transformers-sentiment-analysis
* PyTorch Lightning documentation - https://pytorch-lightning.readthedocs.io/en/0.7.1/