Question answering is a very popular natural language understanding task. It has applications in a wide variety of fields such as dialog interfaces, chatbots, and various information retrieval systems. Answering questions using knowledge graphs adds a new dimension to these fields.

“Question answering over knowledge graphs (KGQA) aims to provide the users with an interface to ask questions in natural language, using their own terminology, to which they receive a concise answer generated by querying the KG.” (Source)

In this blog post, I introduce the KGQA (knowledge graph question answering) task and discuss various methods involved in it. But first, let’s go through a brief description of knowledge graphs.

What is a Knowledge Graph?

A knowledge graph is a graph-based data model that describes real-world entities and relations between them. For example, “Gal Gadot” and “Israel” are entities and “born_in” is the relation between them. An entity can also be linked to a data value, often referred to as literals, e.g., “30 April 1985”.

Let’s look at a formal definition of knowledge graphs. Let E be the set of all entities in a knowledge graph, L be the set of all literals, and P be the set of relations linking an entity to another entity or an entity to a literal. Then, a knowledge graph can be defined as the subset of all possible triples of E -> P -> (E or L).

Publicly Available Knowledge Graphs

Wikidata, Freebase, and DBpedia are some massive publicly available knowledge graphs. These KGs can be queried using standard query languages such as SPARQL. Wikidata contains background information about almost any topic in the world. Here, is a page on Gal Gadot.

Knowledge Graph Question Answering Task

The KGQA task involves answering a natural language question using the information stored in a KG. The input question is first translated into a formal query language e.g., SPARQL and then this formal query is executed over the KG to fetch the answer.

The task of converting a natural question into a formal KG query is called Semantic parsing. Let’s have a look at an example –

Input: “What are all the country capitals in Africa?”

Output:

PREFIX ex: <http://example.com/exampleOntology#>
SELECT ?capital
       ?country
WHERE
  {
    ?x  ex:cityname       ?capital   ;
        ex:isCapitalOf    ?y         .
    ?y  ex:countryname    ?country   ;
        ex:isInContinent  ex:Africa  .
  }

Once we get the output of a semantic parser, the query can easily be executed over the KG. This makes semantic parsing the most important step in a KGQA task and a lot of research has been focused on this. Let’s have a look at some of the techniques involved in this step.

Neural Network Based Semantic Parsers

The prediction models commonly used in semantic parsing can be categorized into – classification, ranking, and translation based models. Let’s have a look at each of these methods in detail.

Classification Based KGQA Models

A semantic parser should be able to generate a target formal query of any arbitrary length. But we will look into these complex queries in detail later. A simpler task than this is when we can assume a fixed simple structure of the output query.

“Gal Gadot was born in Israel” -> <ex: Gal Gadot, ex: bornIn, ex: Israel>

Simple queries comprise one subject entity (Gal Gadot), one relation (born in), and one object entity (Israel). Classification based models tend to work well for such simple queries.

This simple query prediction task can be broken down into two sub-tasks –

• Relation prediction
• Entity Prediction

Relation Prediction

Given a natural question q, relation classification involves predicting one of the n relations of a given KG. This can be treated as a standard text sequence classification problem where the natural question q is the input and one of the n relations is the output label. The input natural question is encoded into a fixed-length vector. The encoder network can be any neural architecture (RNN, CNN, transformer, etc.). The encoder output is fed into a linear layer with dimensions N x d, where N is the number of relations in the KG and d is the size of the encoded vector. The label probabilities can be computed with a standard softmax. 

Entity Prediction

We can employ the same method as above and have two similar models for relation and entity prediction. But, large KGs such as freebase contain a huge amount of entities. While the number of relations in a KG might be in thousands the number of entities can be in millions and training a classifier for millions of output labels is not feasible.

Therefore, a common two-step technique is employed for entity prediction –

• An entity span detector is employed to detect the entity form a natural question text.

“Where was Gal Gadot born?” -> “Gal Gadot”

• Then, a simple text-based retrieval method is used to find a suitable candidate(s) from the KG entities for the identified entity from the question.

The entity span prediction from a natural question can be treated as a classification problem where two separate classifiers predict the start and the end token of the entity. Or it can be treated as a sequence to sequence tagging problem which predicts for every token if it belongs to the entity or not.

Ranking Based KGQA Models

The above classification based methods will not work well for complex queries. Since the set of possible queries of arbitrary length can grow exponentially. Ranking based KGQA models employ a two-step procedure to counter this. First, they employ a search based procedure to find the top few probable query candidates. Then they use a neural network-based ranking model to find the best candidate. 

Probable Candidates Search Procedure

Many techniques can be employed to reduce the probable candidate space, such as –

• Using already existing entity linking tools to limit the search space to only those formal queries that contain entities and relations within a certain distance (hops) of the entities in the input question.
• Selecting formal queries that have a fixed number of relations.
• Using beam search with a scoring mechanism.
• Eliminating invalid or incorrect queries that violate the rules of the formal query language.

Ranking Probable Candidates

Given a natural query and a set of probable target query candidates, the task of ranking involves assigning a score to each of the formal queries. This can be done in a two-step procedure –

• Encoding the natural query and every candidate to a vector space. A simple bag of words can be used for the tokens in both the input questions and the formal query. Neural network models can also be used to learn embeddings for the tokens.
• Calculating the score of every candidate using a scoring mechanism. This scoring mechanism can be a simple dot product or another neural network that takes the two vectors and outputs a score.

Training Ranking Models

Training of these ranking models involves learning parameters for the two encoders and/or the score calculator. This can be done using –

• Full Supervision, where training data consists of natural questions and their corresponding formal query representations. For these datasets, a set of negative examples (a natural question and an incorrect formal query) is generated randomly for every natural question in the dataset. And the model is trained by minimizing a point-wise or pairwise ranking loss (see this).
• Weak Supervision, where training data consists of only pairs of natural questions and corresponding answers (execution results from the KG). For these datasets, a similar search procedure that was used for probable candidates (discussed earlier) is employed to find pseudo gold labels. Then these pseudo gold labels are used for model training similar to full supervision.

Translation Based KGQA Models

In these models, semantic parsing is treated as a translation problem. A sequence-to-sequence model is learned to translate a natural question into a formal query similar to language (machine) translation models. 

Neural sequence to sequence models are commonly used for this. A typical neural seq-to-seq model consists of –

• Encoder – It encodes the input sequence to create context-dependent representations.
• Decoder – It generates the output sequence one token at a time conditioning on the previously generated tokens and the input sequence.

RNNs, CNNs, and transformers are some of the popular encoder-decoder systems.

Constrained Decoding

A simple decoder does not leverage the strict nature of the KG queries. The output sequence that needs to be generated follows strict grammatical rules such that when executed over a KG return valid results. These rules can be leveraged during both training and inference to get better results. For example, in the case of parsing simple questions (entity -> relation -> entity), the decoder can be constrained to generate only entities in the first step, only relations as the second token, followed by only an entity in the third step.

Further Reading

I am very excited to see knowledge graphs gain so much traction of late. If you are interested in digging further, I highly recommend these resources –

• Chakraborty et al., 2019 – A comprehensive study on knowledge graph question answering.
• Knowledge Graphs in Natural Language Processing @ ACL 2019

I hope you enjoyed the post. Given my recent interest, you can expect many more posts on knowledge graphs. Please stay tuned for more :-).

BERT is a deep learning model that has given state-of-the-art results on a wide variety of natural language processing tasks. It stands for Bidirectional Encoder Representations for Transformers. It has been pre-trained on Wikipedia and BooksCorpus and requires task-specific fine-tuning.

What is the model architecture of BERT?

BERT is a multi-layer bidirectional Transformer encoder. There are two models introduced in the paper.

• BERT base – 12 layers (transformer blocks), 12 attention heads, and 110 million parameters.
• BERT Large – 24 layers, 16 attention heads and, 340 million parameters.

For an in-depth understanding of the building blocks of BERT (aka Transformers), you should definitely check this awesome post – The Illustrated Transformers.

What is the flow of information of a word in BERT?

A word starts with its embedding representation from the embedding layer. Every layer does some multi-headed attention computation on the word representation of the previous layer to create a new intermediate representation. All these intermediate representations are of the same size. In the figure above, E1 is the embedding representation, T1 is the final output and Trm are the intermediate representations of the same token. In a 12-layers BERT model a token will have 12 intermediate representations.

What are the tasks BERT has been pre-trained on?

Masked Language Modeling and Next Sentence Prediction.

What is Masked Language Modeling?

Language Modeling is the task of predicting the next word given a sequence of words. In masked language modeling instead of predicting every next token, a percentage of input tokens is masked at random and only those masked tokens are predicted.

Why use masked language modeling over standard language modeling?

Bi-directional models are more powerful than uni-directional language models. But in a multi-layered model bi-directional models do not work because the lower layers leak information and allow a token to see itself in later layers.

How is masked language modeling implemented in BERT?

The masked words are not always replaced with the masked token – [MASK] because then the masked tokens would never be seen before fine-tuning. Therefore, 15% of the tokens are chosen at random and –

• 80% of the time tokens are actually replaced with the token [MASK].
• 10% of the time tokens are replaced with a random token.
• 10% of the time tokens are left unchanged.

What is Next Sentence Prediction?

Next sentence prediction task is a binary classification task in which, given a pair of sentences, it is predicted if the second sentence is the actual next sentence of the first sentence.

This task can be easily generated from any monolingual corpus. It is helpful because many downstream tasks such as Question and Answering and Natural Language Inference require an understanding of the relationship between two sentences.

What downstream tasks can BERT be used for?

BERT can be used for a wide variety of tasks. The two pre-training objectives allow it to be used on any single sequence and sequence-pair tasks without substantial task-specific architecture modifications.

How is the input text represented before feeding to BERT?

The input representation used by BERT is able to represent a single text sentence as well as a pair of sentences (eg., [Question, Answer]) in a single sequence of tokens.

• The first token of every input sequence is the special classification token – [CLS]. This token is used in classification tasks as an aggregate of the entire sequence representation. It is ignored in non-classification tasks.
• For single text sentence tasks, this [CLS] token is followed by the WordPiece tokens and the separator token – [SEP].

• For sentence pair tasks, the WordPiece tokens of the two sentences are separated by another [SEP] token. This input sequence also ends with the [SEP] token.

• A sentence embedding indicating Sentence A or Sentence B is added to each token. Sentence embeddings are similar to token/word embeddings with a vocabulary of 2.
• A positional embedding is also added to each token to indicate its position in the sequence.

Which Tokenization strategy is used by BERT?

BERT uses WordPiece tokenization. The vocabulary is initialized with all the individual characters in the language, and then the most frequent/likely combinations of the existing words in the vocabulary are iteratively added.

How does BERT handle OOV words?

Any word that does not occur in the vocabulary is broken down into sub-words greedily. For example, if play, ##ing, and ##ed are present in the vocabulary but playing and played are OOV words then they will be broken down into play + ##ing and play + ##ed respectively. (## is used to represent sub-words).

What is the maximum sequence length of the input?

512 tokens.

How many layers are frozen in the fine-tuning step?

No layers are frozen during fine-tuning. All the pre-trained layers along with the task-specific parameters are trained simultaneously.

Is discriminative fine-tuning used?

No. All the parameters are tuned with the same learning rate.

What are the optimal values of the hyperparameters used in fine-tuning?

The optimal hyperparameter values are task-specific. But, the authors found that the following range of values works well across all tasks –

• Dropout – 0.1
• Batch Size – 16, 32
• Learning Rate (Adam) – 5e-5, 3e-5, 2e-5
• Number of epochs – 3, 4

The authors also observed that large datasets (> 100k labeled samples) are less sensitive to hyperparameter choice than smaller datasets.

What is the fine-tuning procedure for sequence classification tasks?

The final hidden state of the [CLS] token is taken as the fixed-dimensional pooled representation of the input sequence. This is fed to the classification layer. The classification layer is the only new parameter added and has a dimension of K x H, where K is the number of classifier labels and H is the size of the hidden state. The label probabilities are computed with a standard softmax.

What is the fine-tuning procedure for sentence pair classification tasks?

This procedure is exactly similar to the single sequence classification task. The only difference is in the input representation where the two sentences are concatenated together.

What is the fine-tuning procedure for question answering tasks?

Question answering is a prediction task. Given a question and a context paragraph, the model predicts a start and an end token from the paragraph that most likely answers the question.

Just like sentence pair tasks, the question becomes the first sentence and paragraph the second sentence in the input sequence. There are only two new parameters learned during fine-tuning a start vector and an end vector with size equal to the hidden shape size. The probability of token i being the start of the answer span is computed as – softmax(S . K), where S is the start vector and K is the final transformer output of token i. The same applies to the end token.

What is the fine-tuning procedure for single sentence tagging tasks?

In single sentence tagging tasks such as named entity recognition, a tag must be predicted for every word in the input. The final hidden states (the transformer output) of every input token is fed to the classification layer to get a prediction for every token. Since WordPiece tokenizer breaks some words into sub-words, the prediction of only the first token of a word is considered.

Can BERT be used with tensorflow?

Yes. The official open sourced code is in tensorflow (GitHub).

Can BERT be used with Pytorch?

Yes. Huggingface has open sourced the repository – transformers. It supports the op-to-op implementation of the official tensorflow code in PyTorch and many new models based on transformers.

Can BERT be used with Fastai?

As of now, fastai does not have official support for BERT yet. But, there are ways we can get around with it. This article demonstrates how BERT can be used with fastai.

Can BERT be used with Keras?

Yes. Check this out – BERT-keras.

How to use BERT as a sentence encoder?

The final hidden states (the transformer outputs) of the input tokens can be concatenated and / or pooled together to get the encoded representation of a sentence. bert-as-a-service is an open source project that provides BERT sentence embeddings optimized for production. I highly recommend this article – Serving Google BERT in Production using Tensorflow and ZeroMQ.

How does BERT perform when used as a sentence encoder with a task-specific architecture (similar to ELMO)?

BERT is effective for both fine-tuning and feature-based approaches. The authors did ablation studies on the CoNLL-2003 NER task, in which they took the output from one or more layers without fine-tuning and fed them as input to a randomly initialized two-layer 768 dimensional BiLSTM before the classification layer. The best performing model was the one that took representations from the top four hidden layers of the pre-trained transformer.

Is BERT available in languages other than english?

Yes, there is a multilingual BERT model available as well.

Is BERT available on domain specific pre-trained corpus?

Yes. I have come across Clinical BERT – BERT pre-trained on clinical notes corpus and sciBERT – Pre-Trained Contextualized Embeddings for Scientific Text.

How long does it take to pre-train BERT?

BERT-base was trained on 4 cloud TPUs for 4 days and BERT-large was trained on 16 TPUs for 4 days. There is a recent paper that talks about bringing down BERT pre-training time – Large Batch Optimization for Deep Learning: Training BERT in 76 minutes.

How long does it take to fine-tune BERT?

For all the fine-tuning tasks discussed in the paper it takes at most 1 hour on a single cloud TPU or a few hours on a GPU.

Thanks for reading. Please feel free to suggest more questions in the comment section. Your valuable suggestions are always welcome.

The working of supervised MT systems makes intuitive sense. The input is a source sentence and the output is its translation in the target sentence. But unsupervised MT is at times difficult to grasp especially when they leverage only monolingual corpora of source and target languages. In an attempt to explain the general working of such systems, I will discuss an unsupervised machine translation method that was presented at ICLR 2018.

This model comes from a paper published by Facebook AI Research – Unsupervised Machine Translation Using Monolingual Corpora Only. They propose a method in which the model leverages only monolingual corpora in two different languages and maps sentences from both the languages in a common feature space. From this common feature space, the model is forced to reconstruct sentences in both languages. Let’s call this common feature space latent space from now on.

Before going through the overall working of the model, let’s go through the individual components. The translation model comprises an encoder, a decoder, pre-trained embeddings for both the languages and a discriminator.

Pre-Trained Embeddings

The embeddings for the source and the target language are trained on the two monolingual corpora separately using FastText. Then, a bilingual dictionary is learned using an unsupervised method (Word Translation Without Parallel Data). This can be used for word-by-word translation. Using this procedure, we now have a common set of embeddings for words from both the languages.

Encoder

Given an input sequence of n words in a particular language (source or target), the encoder (a bidirectional-LSTM) computes a sequence of n hidden states by using the pre-trained word embeddings. These hidden states are vectors in the latent space we mentioned earlier. It is important to note that the encoder does not differentiate between the two languages and has a shared set of LSTM layer weights for both. This means that the input sequence of words (in the form of embeddings) can be from any of the two languages and the same encoder converts them into vectors of the latent space.

Decoder

The decoder (also an LSTM) takes as input the sequence of hidden vectors generated by the encoder and outputs a sequence of words in the target language. At each step, the decoder, takes as input the previous hidden state, the current word and a context vector given by a weighted sum over the encoder states. The process is repeated until the decoder generates a stop symbol indicating the end of the sequence.

Similar to the encoder, the decoder also has the same LSTM layer weights for both the languages. The input to the decoder includes a language-specific start symbol to make it generate output in a particular language. The encoder-decoder forms a sequence to sequence model with attention.

Discriminator

The discriminator is a neural network that is trained to classify the hidden vectors (of the latent space) generated by the encoder. The discriminator predicts whether a vector of the latent space is an encoding of a source sentence or a target sentence. It is used to accomplish a particular thing (more on this later).

Now that we have gone through the internal components of the model, let’s see the methods involved in training the full model. There are three steps taking place during model training –

• Denoising Auto-Encoding – The model learns to reconstruct a sentence in a given language from a noisy version of it.
• Cross-Domain Training – The model also learns to reconstruct any source sentence given a noisy translation of the same sentence in the target language, and vice versa.
• Adversarial training – The model is constrained to have the same distribution of the source and target latent vectors.

Denoising Auto-Encoding

In this step, the system is trained to re-create its own input. It takes an input sentence, encodes it, and tries to reconstruct the original sentence in the same language. Since we have a common latent space, this would help in making the encoder and decoder learn how to convert the vectors of the latent space into the source or target sentences.

But, there is a problem with this training procedure. The model is just trying to learn a trivial copying task. Because of which, it may just learn to blindly copy input words in the output, without learning any real knowledge of the languages involved. To counter this, the authors propose the introduction of random noise in the input (similar to stacked denoising autoencoders). They randomly drop a word in the input sentence with some probability and shuffle the input sentence slightly. The system is then asked to reconstruct the original sentence from this noisy version.

The loss minimized in this step can be given as –

L_auto(θ_enc, θ_dec, Z, l) = E[Δ(x, x̂)]

where θ_enc is the encoder parameters, θ_dec is the decoder parameters, Z is the pre-trained embeddings, l is the source or target language, Δ is a measure of discrepancy (sum of token-level cross-entropy losses), x̂ is the reconstructed output of the model (x̂ ∼ decoder(encoder(noise(x), l), l)). The total loss for this step is,

L_auto(θ_enc, θ_dec, Z, l1) + L_auto(θ_enc, θ_dec, Z, l2)

Cross Domain Training

In this step, the system learns to translate the input sentence from one language into the other. The translation model (M) improves in an iterative procedure. We start with an initial model that is a naive word to word translation model from a dictionary learned in an unsupervised way. Let’s call this initial model M_i. 

Now, a sentence is sampled from the source language. It is translated using M_i into the target language.

translated target sentence = M_i(source sentence)

The encoder-decoder system now tries to reconstruct a noisy version of this translated sentence back into the source language sentence by optimizing the following loss function –

L_cd(θ_enc, θ_dec, Z, l1, l2) = E[Δ(x̂, x)]

where θ_enc is the encoder parameters, θ_dec is the decoder parameters, Z is the pre-trained embeddings, l1 is the source language, l2 is the target language, Δ is again the sum of token-level cross-entropy losses, x̂ is the reconstructed output of the model (x̂ ∼ decoder(encoder(noise(translated target sentence), target), source)), and x is the initial source sentence.

The above step is done with both the languages being source and target alternately. The total loss for this step becomes –

Lcd(θenc, θdec, Z, l1, l2) + Lcd(θenc, θdec, Z, l2, l1)

After this, the current translation model is updated with the current encoder-decoder pair to get Mi+1. This process is iteratively repeated improving the encoder-decoder translation system on every step. At inference time, the final encoder-decoder system is directly used for translation.

Intuition: The idea behind the algorithm is that as long as the initial model retains at least some information of the input sentence, the encoder will map such translation to vectors in the latent space that also correspond to a cleaner version of the input because the encoder is trained to denoise. At the same time, the decoder is trained to predict noiseless outputs, given noisy features. Putting these two things together will produce less noisy translations, which will enable better back-translations at the next iteration, and so on so forth.

Adversarial Training

Since we have a shared space of encoded hidden vectors (the latent space) for both the languages, it is important for vectors encoded from the two languages to be not very different from one another (to have same distribution). Otherwise, the decoder will face problems during translation steps. To add such a constraint to the system, a discriminator (discussed earlier) is trained. It predicts the language (source or target) from which a latent space vector has been encoded. The encoder, on the other hand, is trained to fool the discriminator by optimizing –

L_adv(θ_enc, Z | θ_discrimiator) = -Log(Prob(L₁ | encoder(x, L₂)))

where θ_enc is the encoder parameters, Z is the pre-trained embeddings, θ_discrimiator is the discriminator parameters, x is the source sentence.

The discriminator loss is minimized in parallel to update the discriminator.

Final Objective

The final objective function at each step can be given as –

λ_auto [L_auto(θ_enc, θ_dec, Z, l1) + L_auto(θ_enc, θ_dec, Z, l2)]+
λ_cd [Lcd(θenc, θdec, Z, l1, l2) + Lcd(θenc, θdec, Z, l2, l1)] +
λ_adv [L_adv(θ_enc, Z | θ_discrimiator)]

where λ_auto, λ_cd, and λ_adv are hyper-parameters weighting the importance of the auto-encoding, cross-domain and adversarial loss.

Algorithm

Unsupervised Model Selection Criterion

Since there is no parallel data to measure the quality of the translation system, the authors adopt a proxy method of evaluation. All sentences in one language are translated into the other language and then translated back to the original language. The quality of the model is then evaluated by computing the BLEU score over the original inputs and their reconstructions via this two-step translation process. The performance is then averaged over the two directions, and the selected model is the one with the highest average score.

Similar Approaches

I only discussed a particular method for unsupervised translation. But the auto-encoding followed by back translation approach has been used in various other places. Unsupervised Neural Machine Translation is another paper presented at ICLR 2018 that leverages only monolingual corpora. Instead of having a shared encoder and decoder, this system only shares the encoder and has separate decoders for the two languages.

A more recent work – Phrase-Based & Neural Unsupervised Machine Translation has also been published by Facebook AI Research in which the authors highlight three key requirements for unsupervised MT – a good initialization, language modeling, and modeling the inverse task (via back-translation).

Conclusion

While still not able to complete with supervised learning having a large amount of parallel data, unsupervised MT has a lot of promise for low resource languages. I hope there is more work on similar unsupervised methods in the future. Thank You

Leslie N. Smith in his paper – A Disciplined Approach to Neural Network Hyper-Parameters: Part 1 – Learning Rate, Batch Size, Momentum, and Weight Decay discusses several efficient ways to set the hyper-parameters in a neural network aimed at reducing training time and improving performance. In this blog post, I summarize the various strategies recommended in the paper. All the experiment figures have been directly taken from the paper.

Validation Loss Is Important

Validation/Test (have been used interchangeably) loss is very important to identify clues for overfitting and underfitting. The author suggests that identifying these clues early during training can allow us to tune the hyper-parameters with only a few epochs. This helps in avoiding complete grid or random searches. 

For example, the above figure shows plots of the training loss, test loss and test accuracy for a learning rate range test (discussed later) of a residual network on the Cifar dataset to find suitable values of learning rate for training.  We can see that in the learning rate range of 0.01-0.04, the test loss within the black box indicates overfitting (test loss is increasing). This information is not present in the other two curves. Now we know that this architecture has the capacity to overfit and a small learning rate will cause overfitting. 

The key takeaway from this section is that achieving the horizontal part of the test loss is the goal of hyperparameter tuning. The test loss can be used to find the optimal hyper-parameters without performing a full training. If the hyper-parameters are set well at the beginning they will perform well through the entire training process. 

Next, I will summarize the various tips and strategies provided by the author to identify the optimal values of – learning rate, batch size, momentum, and weight decay.

Learning rate

Learning rate is perhaps the most important hyper-parameter in neural network training. If the learning rate is too small, overfitting can occur. Large learning rates regularize the training but a very high value can cause the training to diverge. The key takeaways from this section would be –

• Using a large learning rate can lead to superconvergence.
• Use a Learning rate range test to find a large learning rate.
• 1cycle policy should be used with the LR range test.
• The total amount of regularization must be balanced.

Let’s discuss each of the above in detail.

Learning Rate Range Test

Instead of using a single value for the learning rate, the author proposes the use of learning rate range test. In this method, training is started from a very low value and is slowly increased to a large value until the loss starts to diverge. Test loss vs. learning rate is plotted and a value before the minimum test loss (where the loss still improves) is selected as the maximum value. The minimum value is ten times lower. 

1cycle Policy

The author advises the use of 1cylce policy to vary the learning rate between this range. He recommends to do a cycle with two steps of equal lengths, one going from a lower learning rate to a higher one then going back to the minimum.  The length of this cycle should be slightly less than the total number of epochs, and, in the last part of training, the learning rate is decreased more than the minimum, by several orders of magnitude.

Both these techniques were also introduced by Leslie in his paper – Cyclical Learning Rates for Training Neural Networks. For a detailed explanation of the 1cycle policy and the LR range test, check this awesome article by Sylvian Gugger –  The 1cycle policy

Superconvergence is a phenomenon where neural networks can be trained an order of magnitude faster than with standard training methods. The above figure is an example of superconvergence where training was completed in 10000 iterations with a 1cycle learning rate policy where learning rate went up to 3.0 instead of needing 80000 iterations with a fixed learning rate of 0.1.

Balance The Amount of Regularization

The author suggests that the amount of regularization must be balanced for each dataset and architecture. If we are using large learning rates (which have a significant underfitting effect), we must reduce other forms of regularizations. 

The above figure shows that higher weight decay values do not go well with a higher learning rate (i.e. up to 3). Since a higher weight decay value provides a greater regularization effect, a balance in the amount of regularization is disturbed when used with a large learning rate which also has a regularization effect.

Batch Size

The batch size can also affect the underfitting and overfitting balance. Smaller batch sizes provide a regularization effect. But the author recommends the use of larger batch sizes when using the 1cycle policy.

Instead of comparing different batch sizes on a fixed number of iterations or a fixed number of epochs, he suggests the comparison should be done with a constant execution time. Since our goal is to maximize performance while minimizing the computational execution time.

The above figure shows the validation accuracy for four different batch sizes. All the runs had the same execution time. We can see that larger learning rates were possible with higher batch sizes. The small black box gives a magnified view to highlight the difference in the accuracies. The results imply that it is beneficial to use large batch sizes. An important note – It was also found that unlike the final accuracy values the final loss values were lower for smaller batch sizes. Despite this, the paper recommends the use of a batch size that fits in our hardware’s memory and enable using larger learning rates.

Momentum

Momentum and learning rate are closely related. The optimal values of these hyper-parameters are interdependent. Momentum is designed to accelerate network training and has a similar impact on the weight updates as the learning rate. Like learning rates, it is valuable to set momentum as large as possible without causing instabilities during training. 

Experiments show that unlike the learning rate range test, a momentum range test is not useful for finding an optimal momentum. The author recommends that short runs of 0.99, 0.97, 0.95, and 0.9 will quickly show the best value for momentum.

With cyclic learning rates, it is better to use a cyclical momentum that starts at the maximum momentum (one of the above values) and keeps decreasing to a value of 0.8 or 0.85 as the learning rate increases. The intuition behind this is that in this part of the training we would want to quickly go in new directions to find flatter areas, so the new gradients need to be given more weight. The momentum is then increased to the maximum value as the learning rate decreases in the second half of the 1cycle.

If a constant learning rate is used, then a large constant momentum (0.9 – 0.99) will help in speeding up the training. According to the author, the exact best value of momentum chosen during the whole training can give us the same final results, but using cyclical momentums helps us avoid multiple values running a number of cycles.

Weight Decay

Weight decay is a type of regularization method and is a key component during training. We have already discussed that we must balance the total amount of regularization during training. Hence, we need to set the values for weight decay properly. 

The key takeaways from this section of the paper are –

• Cyclic weight decay is not helpful. Experiments show that weight decay is not like learning rates or momentum and the best value should remain constant throughout the training.
• A grid search is worthwhile and any difference due to the weight decay value is visible early in the training.
• If we have no idea of a reasonable value for weight decay, we should test 10^-3, 10^-4, 10^-5, and 0.
• Smaller datasets and architectures seem to require larger values for weight decay while larger datasets and deeper architectures seem to require smaller values.
• The optimal weight decay is different if we search with a constant learning rate versus using a learning rate range test.

In the above experiment, a learning rate range test (LR=0.001 – 0.01) along with decreasing momentum (= 0.98 – 0.8) has been done with different weight decay values. It shows that a weight decay value of 1.8 x 10^-3 is best as it remains stable for larger learning rates and also attains a lower validations loss.

Conclusion

The paper deals with perhaps the most common problem deep learning practitioners have – hyper-parameter tuning. I believe this is a must-read paper. I hope you enjoyed reading my post and now it would be easier if you go through the original paper. Thanks a lot.

In this blog post, I go through the research paper – Regularizing and Optimizing LSTM Language Models that introduced the AWD-LSTM and try to explain the various techniques discussed in it. The paper investigates a set of regularization and optimization strategies for word-based language modeling tasks that are not only highly effective but which can also be used with no modification to existing LSTM implementations.

The Chinese version of the post, translated by Jakukyo Friel (@weakish) can be found here – 语言建模的王者：AWD-LSTM指南

The AWD-LSTM stands for ASGD Weight-Dropped LSTM. It uses DropConnect and a variant of Average-SGD (NT-ASGD) along with several other well-known regularization strategies. We will go through all these techniques in detail. While all these methods have been proposed and theoretically explained before, the beauty of this paper lies in their successful application to the language modeling task achieving state-of-the-art results.

The code for reproducing the results is open sourced and is available at the awd-lstm-lm GitHub repository.

If you want to refresh your memory with the internal working of an LSTM network you should definitely check out this famous article – Understanding LSTM Networks by Christopher Olah.

The mathematical formulation of an LSTM can be expressed as –

i_t  = σ(Wⁱx_t  + Uⁱh_t-1)
f_t = σ(W^fx_t  + U^fh_t-1)
o_t = σ(W^ox_t  + U^oh_t-1)
c’_t = tanh(W^cx_t  + U^ch_t-1)
c_t = i_t ⊙ c’_t + f_t ⊙ c’_t-1
h_t = o_t ⊙ tanh(c_t)

where, Wⁱ, W^f, W^o, W^c, Uⁱ, U^f, U^o, U^c are weight matrices, x_t is the vector input to timestep t, h_t is the current exposed hidden state, c_t is the memory cell state, and ⊙ is element-wise multiplication.

Let’s go through each of the strategies proposed by the authors one by one.

Weight-dropped LSTM

The recurrent connections of an RNN have been prone to overfitting. Preventing this has been an area of great interest and extensive research. In this regard, Dropouts have been massively successful in feed-forward and convolutional neural networks. But applying dropouts similarly to an RNN’s hidden state is ineffective as it disrupts the RNN’s ability to retain long-term dependencies.

There have been various strategies to overcome this problem but the majority of previous techniques act either on the hidden state vector h_t-1 or on the update to the memory state c_t. This prevents the use of inflexible but highly optimized black box RNN implementations such as the NVIDIA’s cuDNN LSTM.

To counter this problem, the authors propose the use of DropConnect. We know that in Dropout, a randomly selected subset of activations is set to zero within each layer. In DropConnect, instead of activations, a randomly selected subset of weights within the network is set to zero.  Each unit thus receives input from a random subset of units in the previous layer.

The DropConnect is applied on the hidden to hidden weight matrices (Uⁱ, U^f, U^o, U^c) instead of the hidden or memory states. Since this dropout operation is performed once, before the forward and backward pass the impact on training speed is minimal and any standard optimized black box RNN implementation can be used. By performing dropout on the hidden-to-hidden weight matrices, overfitting can be prevented on the recurrent connections of the LSTM.

You can have a look at the weight_drop.py module from the official awd-lstm-lm Github repository for a detailed implementation.

The authors suggest that DropConnect can also be used on non-recurrent weights of the LSTM. Though the focus was on preventing overfitting on the recurrent connections.

Non-monotonically Triggered Average SGD

It has been found that for the particular task of language modeling, traditional SGD without momentum outperforms other algorithms such as momentum SGD, Adam, Adagrad, and RMSProp. Therefore, the authors investigate a variant of the traditional SGD algorithm known as ASGD (Average SGD).

Average SGD

Let’s first go through the ASGD algorithm. It takes gradient update step identical to the SGD algorithm but instead of returning the weight computed in the current iteration, it also takes into account the weights of the previous iterations and returns an average.

Traditional SGD update –

w_t = w_prev - lr_t * grad(w_prev)

ASGD update –

avg_fact = 1 / max(t - K, 1)

if avg_fact != 1:
  w_t = avg_fact * (sum(w_prevs) + (w_prev - lr_t * grad(w_prev)))
else:
  w_t = w_prev - lr_t * grad(w_prev)

where,

K is the minimum number of iterations run before weight averaging starts. So before K iterations, the ASGD will behave similarly to a traditional SGD. t is the current number of iterations done, sum(w_prevs) is the sum of weights from iteration K to t and lr_t is the learning rate at iteration t decided by a learning rate scheduler.

You can find the PyTorch implementation of AGSD here.

The authors highlight two drawbacks of this method –

• Unclear tuning guidelines to the learning rate scheduler.
• Unclear guidelines on the value of the parameter K. A value too small can adversely affect the efficacy of the method and a value too large may take additional iterations to converge.

They propose a non-monotonically triggered variant of ASGD (NT-ASGD) in which-

• Averaging is triggered only when the validation metric fails to improve for multiple cycles. This is ensured by the non-monotone interval hyperparameter n. So whenever the validation metric does not improve for n cycles the algorithm switches to use ASGD. The authors found that setting n=5 works well.
• A constant learning rate is used throughout the experiment and hence no further tuning is required.

The authors use this setting of NT-ASGD and thus demonstrate that it achieves better results as compared to SGD.

Extended Regularization Techniques

In addition to the two techniques discussed above, the authors use additional regularization techniques that prevent overfitting and improve data efficiency.

Variable Length Backpropagation Sequences

The authors highlight the inefficiency of using fixed length backpropagation through time (BPTT). Consider, having 100 elements to perform backpropagation with a fixed BPTT window of 10. In this case, any element divisible by 10 will not have any element to backprop into. This prevents 1/10 of the data to improve itself in a recurrent fashion and 8/10 of the data only uses partial BPTT window.

To counter this, the authors propose the use of variable length backpropagation sequences. They first select the base sequence length to be bptt with probability p and bptt/2 with probability 1-p. In the PyTorch implementation, the authors use p = 0.95.

base_bptt = bptt if np.random.random() < 0.95 else bptt / 2

Then, this base_bptt is used to get the seq_len using N(base_bptt, s) where s is the standard deviation and N is a normal distribution. This is suggested in the paper. Although, the PyTorch implementation has the following code for this step –

seq_len = max(5, int(np.random.normal(base_bptt, 5)))

The learning rate is also rescaled according to the seq_length being used. The rescaling step is necessary as sampling arbitrary sequence lengths with a fixed learning rate favors short sequences over longer ones.

lr2 = lr * seq_len / bptt

Variational Dropout

In a standard dropout, a new dropout mask is sampled each time the dropout connection is called. While in Variational Dropout, the dropout mask is sampled only once upon the first call and then that locked dropout mask is repeatedly used for all connections within the forward and backward pass.

While DropConnect is used rather than Variational Dropout to regularize the hidden-to-hidden transition within an RNN, Variational Dropout is used for all other dropout operations, specifically using the same dropout mask for all inputs and outputs of the LSTM within a given forward and backward pass.  Each example within the mini-batch uses a unique dropout mask, rather than a single dropout mask being used over all examples, ensuring diversity in the elements dropped out.

The variational dropout implementation from the official awd-lstm-lm GitHub repository can be found here. For more details, please refer to the original paper.

Embedding Dropout

The authors employ Embedding Dropout which was first introduced in the paper – A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. It performs dropout on the embedding matrix at a word level and is used for a full forward and backward pass. This results in the disappearance of all occurrences of a specific word within that pass.

Reduction in Embedding Size

The easiest reduction in total parameters for a language model can be achieved by reducing the word vector size. Even though it does not help in reducing overfitting, the size of the embedding dimensions is reduced. The first and last LSTM layers are modified such that their input and output dimensions respectively are equal to the reduced embedding size.

Activation Regularization

L₂-regularization is often used on the weights to reduce overfitting. L₂ decay can also be used on the individual unit activations. This is termed activation regularization. AR penalizes activations that are significantly larger than 0 as a means of regularizing the network.

loss = loss + alpha * dropped_rnn_h.pow(2).mean()

Temporal Activation Regularization

In this, L₂ decay is applied to the difference in outputs of an RNN at different time steps. It penalizes the model from producing large changes in the hidden state.

loss = loss + beta * (rnn_h[1:] - rnn_h[:-1]).pow(2).mean()

Here, alpha and beta are scaling coefficients. AR and TAR loss are only applied to the output of the final RNN layer.

Model Analysis

The authors also experimented with different variants of the best performing model. They removed each of the strategies one at a time to see their impact independently.

Each row in the above figure represents the effect on the perplexity score when that particular strategy is removed. This lets us compare the impact of the various strategies employed independently. For example, the most extreme perplexity jump was in removing the hidden-to-hidden LSTM regularization provided by the weight-dropped LSTM (11 points).

Important Experiment Details

Data – The evaluation of the impact of these approaches was done on two datasets – Penn Tree Bank (PTB) Dataset and WikiText 2 Dataset.

Network Architecture – All experiments use a three-layer LSTM model with 1150 units in the hidden layer and an embedding of size 400.

Batch Size – Batch size was 80 for WT2 and 40 for PTB dataset. It was observed empirically that larger batch sizes (e.g., 40-80) outperformed smaller sizes (e.g., 10-20).

Pointer Models –  The authors also demonstrate the use of pointer based attention models along with these techniques. The neural cache model further improves the perplexity score.

The details of the other hyperparameters can be referred from the paper. The hyperparameters were chosen on trial and error and further improvements may be possible if a fine-grained hyperparameter search is employed.

Conclusion

The paper beautifully explains the use of already existing regularization and optimization strategies on language models. This to me seems a must read for any NLP researcher and even beginners. The authors also highlight that while these strategies are demonstrated on the task of language modeling, they should be generally applicable across other sequence learning tasks. This opens up many possibilities for further research.

Hope you had a fun time and the post makes understanding the actual paper easier. Thanks a lot.

(Edit) A big thanks to Stephen Merity for his shout out -

While word embeddings have been massively popular and successful, efforts to obtain embeddings of larger chunks of text (like sentences) have not been as successful. Approaches to learning sentence embeddings using unsupervised methods (like SkipThought or FastSent) have not given satisfactory results.

In this blog post, I will give you a walkthrough of a recent paper (Supervised Learning of Universal Sentence Representations from Natural Language Inference Data) published by the Facebook research team that employs a supervised method to learn sentence embeddings.

Introduction

The authors describe solutions to two important questions that arise in building a sentence embedding model – the type of task to be used for training the sentence vectors and the preferable neural network architecture to use to generate sentence encodings. 

The authors, first, generate sentence vectors using a sentence encoding architecture and word vectors as input. This encoder is then followed by a classifier that takes the encoded sentences as input and trains the sentence vectors.

They show that sentence embeddings generated from models trained on a natural language inference classifier (more details below) give best results in terms of accuracy on downstream tasks. They also explore a number of different architectures for sentence encoding. Namely, standard recurrent models such as LSTMs and GRUs, a self-attentive network and a hierarchical convolutional network. We will go through each of these implementations in detail.

The Natural Language Inference task

Natural language inference, also known as recognizing textual entailments aims at finding a directional relationship between text fragments. In this framework, the entailing and entailed texts are termed text (t) and hypothesis (h), respectively. And, “t entails h” (t => h) if, typically, a human reading t would infer that h is most likely true. (Source).

For this task, the authors use the SNLI (Stanford Natural Language Inference) dataset. It consists of 570k human generated English sentence pairs, manually labeled with one of the three categories – entailment, contradiction and neutral. The authors believe that the semantic nature of this task makes it a good candidate to learn sentence representations that capture universally useful features.

The Neural Network Architecture

There are two parts of the neural network architecture.

• The first part is a shared encoder that encodes both the sentences into vectors.
• The second part takes as input the sentence vectors generated in the first part and predicts one of the three labels using a 3-class classifier.

Let’s go through both the parts of the network in detail.

Sentence Encoder Architectures

The authors explore 7 different architectures for sentence encoding. Let’s go through these networks one by one.

LSTM and GRU

These are the simplest encoders used. As we know, we get n number of hidden representations (vectors) for a sequence of n words in an LSTM or GRU network. A sentence, in this case, is represented by the last hidden vector.

The authors use another variant of this method, called BiGRU-last. In this method, the last hidden state of a forward GRU and the last hidden state of a backward GRU is concatenated to form the sentence vector.

BiLSTM with mean/max pooling

For a sequence of n words, a bi-directional LSTM computes a set of n vectors and each vector is a concatenation of a forward LSTM and a backward LSTM that read the sentence in opposite directions. The authors employ two strategies – selecting maximum value over each dimension of the hidden units (max-pooling) or by considering the average of each dimension (mean-pooling). 

Self-attentive network

The self-attentive sentence encoder uses an attention mechanism over the hidden states of a BiLSTM to generate a representation of the sentence vector. This method uses the context vectors which are generated using a weighted linear combination of the hidden vectors. The self-attentive network has multiple views of the input sentence and we get 4 context vectors. The final sentence representation is the concatenation of these 4 context vectors.

If you are interested in knowing the details of the attention mechanism, here is an awesome introduction by Andrew Ng.

Hierarchical ConvNet

The hierarchical convolutional network introduced comprises 4 convolutional layers. At every layer, max-pooling of the feature maps is done to obtain a representation. The final sentence embedding is represented by a concatenation of the 4 max-pooled representations.

NLI Classifier Architecture

Now that we have discussed the various sentence encoding architectures used in the paper, let’s go through the part of the network which takes these sentence embeddings and predicts the output label.

After the sentence vectors are fed as input to this model, 3 matching methods are applied to extract relations between the text, u and hypothesis, v –

• concatenation of the two representations (u, v)
• element-wise product u * v
• and, absolute element-wise difference |u – v |

The resulting vector captures information from both the text, u and the hypothesis, v, and is fed into a 3-class classifier consisting of multiple fully connected layers followed by a softmax layer.

Training Details

• Stochastic gradient descent (SGD) is used with a learning rate of 0.1 and a weight decay of 0.99.
• At each epoch, the learning rate is divided by 5 if the dev accuracy decreases.
• Mini-batches of size 64 is used, and training is stopped when learning rate goes beyond the threshold of 1e-5.
• For the classifier, multi-layer perceptron with 1 hidden-layer of 512 hidden units is used.
• GloVe vectors have been used for pre-trained word embeddings. These vectors have been trained on Common Crawl 840B with 300 dimensions. A more recent version of InferSent, know as InferSent2 has been trained on fastText vectors.

All these details are the same for all the models trained on SNLI.

Evaluation of Sentence Embeddings

The authors evaluate the quality of the sentence representation by using them as features in 12 different transfer tasks. They have also constructed a sentence evaluation tool called SentEval to automate evaluation of all the tasks mentioned in the paper. The tool uses Adam to fit a logistic regression classifier, with batch size 64.

Results

Impact of the sentence encoding architectures

The authors found that different models trained on the same NLI corpus lead to different transfer task results. The BiLSTM with the max-pooling operation performs best on both SNLI and transfer tasks. The transfer task accuracy has been measured by taking the micro and macro averages of the 12 transfer tasks.

Better performance on the training NLI task is not always followed by a better performance on the transfer task.

The authors hypothesize that some models are likely to over-specialize and adapt too well to the biases of a dataset without capturing general-purpose information of the input sentence. The difference between the results seems to come from the different
abilities of the models to incorporate general information while not focusing too much on specific features useful for the task at hand.

Optimization Algorithm

The transfer quality was also found to be influenced by the optimization algorithm. They found that the BiLSTM-max converged faster on SNLI (5 epochs instead of 10), but obtained worse results on the transfer task.

Embedding Size

Increasing embedding sizes lead to increased performance for almost all the models. This was particularly true for models like – BiLSTM-Max, HConvNet, inner-att.

Comparison with SkipThought

The best performing sentence encoder before InferSent is the SkipThought-LN model. It has been trained on a large corpus of ordered sentences (64 million sentences). The InferSent model outperforms the results obtained by the SkipThought vectors.

NLI as a supervised training set

The findings indicate that the model trained on SNLI obtains much better overall results than models trained on other supervised tasks such as COCO, dictionary definitions, NMT, PPDB, and SST.

Some Useful Resources

• The InferSent encoder is publicly available. You can find it here.
• If you are interested in the current state of the art methods of word and sentence embeddings, you should definitely check out this awesome post – The Current Best of Universal Word Embeddings and Sentence Embeddings.
• If you want some background on word embedding developments before 2017, you should have a look at this – Word embeddings in 2017.
• If you are interested in transfer learning, you should check out the recent work of Jeremy Howard and Sebastian Ruder on ULMFiT (here and here).

I hope you found the post useful. Thanks a lot.

If you want to understand the underlying concepts of using categorical feature embeddings, you should definitely check out this awesome post – An Introduction to Deep Learning for Tabular Data. I also did a deep dive in fastai’s tabular module to come up with this network.

PyTorch provides an excellent abstraction in the form of torch.util.data.Dataset. Such dataset classes are handy as they allow treating the dataset as just another iterator object. We will create a class named TabularDataset that will subclass torch.util.data.Dataset. Each iteration of an object of this class will return a list of three elements – the output sample value, a numpy array of continuous features, and a numpy array of categorical features of the sample.

from torch.utils.data import Dataset, DataLoader


class TabularDataset(Dataset):
  def __init__(self, data, cat_cols=None, output_col=None):
    """
    Characterizes a Dataset for PyTorch

    Parameters
    ----------

    data: pandas data frame
      The data frame object for the input data. It must
      contain all the continuous, categorical and the
      output columns to be used.

    cat_cols: List of strings
      The names of the categorical columns in the data.
      These columns will be passed through the embedding
      layers in the model. These columns must be
      label encoded beforehand. 

    output_col: string
      The name of the output variable column in the data
      provided.
    """

    self.n = data.shape[0]

    if output_col:
      self.y = data[output_col].astype(np.float32).values.reshape(-1, 1)
    else:
      self.y =  np.zeros((self.n, 1))

    self.cat_cols = cat_cols if cat_cols else []
    self.cont_cols = [col for col in data.columns
                      if col not in self.cat_cols + [output_col]]

    if self.cont_cols:
      self.cont_X = data[self.cont_cols].astype(np.float32).values
    else:
      self.cont_X = np.zeros((self.n, 1))

    if self.cat_cols:
      self.cat_X = data[cat_cols].astype(np.int64).values
    else:
      self.cat_X =  np.zeros((self.n, 1))

  def __len__(self):
    """
    Denotes the total number of samples.
    """
    return self.n

  def __getitem__(self, idx):
    """
    Generates one sample of data.
    """
    return [self.y[idx], self.cont_X[idx], self.cat_X[idx]]

Let’s move on to model building. Our model will be a simple feed-forward neural network with two hidden layers, embedding layers for the categorical features and the necessary dropout and batch normalization layers.

The nn.Module class is the base class for all neural networks in PyTorch. Our model, FeedForwardNN will subclass the nn.Module class. In the __init__ method of our class, we will initialize the various layers that will be used in the model and the forward method would define the various computations performed in the network.

import torch
import torch.nn as nn
import torch.nn.functional as F


class FeedForwardNN(nn.Module):

  def __init__(self, emb_dims, no_of_cont, lin_layer_sizes,
               output_size, emb_dropout, lin_layer_dropouts):

    """
    Parameters
    ----------

    emb_dims: List of two element tuples
      This list will contain a two element tuple for each
      categorical feature. The first element of a tuple will
      denote the number of unique values of the categorical
      feature. The second element will denote the embedding
      dimension to be used for that feature.

    no_of_cont: Integer
      The number of continuous features in the data.

    lin_layer_sizes: List of integers.
      The size of each linear layer. The length will be equal
      to the total number
      of linear layers in the network.

    output_size: Integer
      The size of the final output.

    emb_dropout: Float
      The dropout to be used after the embedding layers.

    lin_layer_dropouts: List of floats
      The dropouts to be used after each linear layer.
    """

    super().__init__()

    # Embedding layers
    self.emb_layers = nn.ModuleList([nn.Embedding(x, y)
                                     for x, y in emb_dims])

    no_of_embs = sum([y for x, y in emb_dims])
    self.no_of_embs = no_of_embs
    self.no_of_cont = no_of_cont

    # Linear Layers
    first_lin_layer = nn.Linear(self.no_of_embs + self.no_of_cont,
                                lin_layer_sizes[0])

    self.lin_layers =\
     nn.ModuleList([first_lin_layer] +\
          [nn.Linear(lin_layer_sizes[i], lin_layer_sizes[i + 1])
           for i in range(len(lin_layer_sizes) - 1)])
    
    for lin_layer in self.lin_layers:
      nn.init.kaiming_normal_(lin_layer.weight.data)

    # Output Layer
    self.output_layer = nn.Linear(lin_layer_sizes[-1],
                                  output_size)
    nn.init.kaiming_normal_(self.output_layer.weight.data)

    # Batch Norm Layers
    self.first_bn_layer = nn.BatchNorm1d(self.no_of_cont)
    self.bn_layers = nn.ModuleList([nn.BatchNorm1d(size)
                                    for size in lin_layer_sizes])

    # Dropout Layers
    self.emb_dropout_layer = nn.Dropout(emb_dropout)
    self.droput_layers = nn.ModuleList([nn.Dropout(size)
                                  for size in lin_layer_dropouts])

  def forward(self, cont_data, cat_data):

    if self.no_of_embs != 0:
      x = [emb_layer(cat_data[:, i])
           for i,emb_layer in enumerate(self.emb_layers)]
      x = torch.cat(x, 1)
      x = self.emb_dropout_layer(x)

    if self.no_of_cont != 0:
      normalized_cont_data = self.first_bn_layer(cont_data)

      if self.no_of_embs != 0:
        x = torch.cat([x, normalized_cont_data], 1) 
      else:
        x = normalized_cont_data

    for lin_layer, dropout_layer, bn_layer in\
        zip(self.lin_layers, self.droput_layers, self.bn_layers):
      
      x = F.relu(lin_layer(x))
      x = bn_layer(x)
      x = dropout_layer(x)

    x = self.output_layer(x)

    return x

After creating the network architecture we have to run the training loop. For the purpose of demonstration, I am using the dataset from the Kaggle competition – House Prices: Advanced Regression Techniques.

>>> import pandas as pd
>>> import numpy as np
>>> # Using only a subset of the variables.
>>> data = pd.read_csv("train.csv", usecols=["SalePrice", "MSSubClass", "MSZoning", "LotFrontage", "LotArea",
                                         "Street", "YearBuilt", "LotShape", "1stFlrSF", "2ndFlrSF"]).dropna()

We need to instantiate an object of the TabularData class we created earlier. But before that, we need to label encode the categorical features. For this, we will be using sklearn.preprocessing.LabelEncoder.

>>> categorical_features = ["MSSubClass", "MSZoning", "Street", "LotShape", "YearBuilt"]
>>> output_feature = "SalePrice"

>>> from sklearn.preprocessing import LabelEncoder
>>> label_encoders = {}
>>> for cat_col in categorical_features:
        label_encoders[cat_col] = LabelEncoder()
        data[cat_col] = label_encoders[cat_col].fit_transform(data[cat_col])

Let’s instantiate an object of the TabularDataset class.

>>> dataset = TabularDataset(data=data, cat_cols=categorical_features,
                             output_col=output_feature)

In order to run the training loop, we need to create a torch.util.data.Dataloader object. It serves the following purpose –

• creates batches from the dataset
• shuffles the data
• loads the data in parallel

>>> batchsize = 64
>>> dataloader = DataLoader(dataset, batchsize, shuffle=True, num_workers=1)

Now that we have created the basic data structure to run the training loop, we need to instantiate a model object of the FeedForwadNN class created earlier. This class requires a list of tuples, where each tuple represents a pair of total and the embedding dimension of a categorical variable.

>>> cat_dims = [int(data[col].nunique()) for col in categorical_features]
>>> cat_dims
[15, 5, 2, 4, 112]
>>> emb_dims = [(x, min(50, (x + 1) // 2)) for x in cat_dims]
>>> emb_dims
[(15, 8), (5, 3), (2, 1), (4, 2), (112, 50)]

The number of continuous features used is 4. The hidden layer dimension is 50 and 100 for the first and second layers respectively. The embedding dropout used is 0.04. The hidden layer dropouts are 0.001 and 0.01.

>>> device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”
>>> model = FeedForwardNN(emb_dims, no_of_cont=4, lin_layer_sizes=[50, 100],
                          output_size=1, emb_dropout=0.04,
                          lin_layer_dropouts=[0.001,0.01]).to(device)

Finally, let’s run the training loop –

>>> no_of_epochs = 5
>>> criterion = nn.MSELoss()
>>> optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
>>> for epoch in range(no_of_epochs):
      for y, cont_x, cat_x in dataloader:
          
        cat_x = cat_x.to(device)
        cont_x = cont_x.to(device)
        y  = y.to(device)

        # Forward Pass
        preds = model(cont_x, cat_x)
        loss = criterion(preds, y)

        # Backward Pass and Optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Hope you found this useful. You can also find the entire code on my GitHub repository – pytorch-tabular. Thank You.

In this blog post, I will explain the step-by-step working of the method proposed in this paper.

ULMFiT uses a regular LSTM (without any attention short-cut connection or any other complex addition) which is the state-of-the-art language model architecture (AWD-LSTM, Merity et al., 2017a). The LSTM network has 3 layers. (Edit: You can find my blog post giving a walkthrough of the AWD-LSTM paper here – What makes the AWD-LSTM great?) This single architecture is used throughout – for pre-training as well as for fine-tuning.

There are three stages in ULMFiT  –

• General domain language model pre-training
• Target task language model fine-tuning
• Target task classifier fine-tuning

We will discuss each of these stages in detail.

General Domain Language Model Pre-Training

This step is analogous to the ImageNet pre-training in computer vision. Since language modeling (next word prediction) can capture general properties of a language it serves as an ideal source task for pre-training a network. The network is pre-trained on Wikitext-103 (Merity et al., 2017b). It consists of 28,595 preprocessed English Wikipedia articles and 103 million words.

Target Task Language Model Fine-Tuning

Instead of directly using the Wikipedia pre-trained model and fine-tuning it for the classification task, there is an intermediate step involved. In this step, the language model is fine-tuned on data from the target task (on which classification will be performed). This step improves the classification model (discussed in the third step) on small datasets. The paper proposes the following methods to achieve this step-

Slanted Triangular Learning Rates
The target text (here, the text on which classification needs to be done) has a different distribution than the one on which our language model has been pre-trained. The goal for fine-tuning in this stage is to make the model parameters adapt to the task-specific text features. For this, the authors propose slanted triangular learning rates (STLR) in which, the learning rate first increases linearly and then decays linearly. STLR is a modification of the triangular learning rates (Smith 2017) with a short increase and a long decay period.

Intuition – An initial short increase in the learning rate is required because we want the model to quickly converge to a suitable region of the parameter space for the target task. This is followed by a long decay period which allows for the further refining of the parameters. Hence, using the same or an annealed (decaying) learning rate would not be suitable.

The update schedule for the STLR can be given as –

where,

• T is the number of training iterations (where one training iteration is equal to the number of epochs times the number of updates per epoch).
• cut_frac is the fraction of iterations we increase the LR.
• cut is the iteration when we switch from increasing to decreasing the LR.
• for t < cut, p is the number of iterations the LR has increased upon the total number of increasing iterations and, for t >= cut, p is the total number of iterations the LR has decreased upon the total number of decreasing iterations.
• ratio specifies how much smaller the lowest LR is from the maximum LR, η_max.
• η_t is the learning rate at iteration t.
• generally, cut_frac = 0.1, ratio = 32 and η_max = 0.01

Discriminative Fine-Tuning
Jeremy first introduced discriminative fine tuning as differential learning rates in his course – fast.ai, Part1 2018, Lecture 1. The basic principle behind this is that different layers in a model capture different types of information and hence require different learning rate. The initial layers capture the most general form of information. Here is an awesome visualization toolkit on this.

Similarly, in a language modeling task, the first few layers would have the most general information of the language and would require the least amount of fine-tuning (change in their weights). Subsequently, as we move towards the last layer the amount of required fine-tuning would increase.

Therefore, instead of using the same learning rate in the entire model, the paper proposes to use different learning rates for each layer. It first chooses the learning rate of the last layer by fine-tuning only the last layer and uses the following formula for the lower layers –

 η^l-1 = η^l / 2.6, where η^l is the learning rate of the l-th layer.

Target Task Classifier Fine-Tuning

The final stage in ULMFiT involves training the model with two additional linear blocks. ReLU activation is used for the intermediate layer and softmax for the final linear layer. Each block uses batch normalization and dropout.

Concat Pooling
The first linear layer takes as input the pooled last hidden layer states. The signal in a text classification task is often contained in a few words, which may occur anywhere in the document. Hence, information may get lost if we only consider the last hidden state of the model. Therefore, the last hidden state of the model is concatenated with the max-pooled and the mean-pooled representations of the hidden states over as many time step as fit in the GPU memory.

Gradual Unfreezing
The target classifier fine-tuning is very sensitive and an aggressive fine-tuning may lead to nullifying the benefits of language model pre-training. Therefore, the authors propose gradual unfreezing for fine-tuning the classifier.

• The last LSTM layer is first unfrozen and the model is fine-tuned for one epoch.
• Then the next lower frozen layer is unfrozen.
• This process is repeated until all layers are fine-tuned to convergence.

BPTT for Text Classification (BPT3C)
Similar to backpropagation through time for language modeling, BPTT for text classification is introduced. The document is divided into fixed-length chunks. At the beginning of each chunk, the model is initialized with the final state of the previous chunk. Gradients are back propagated to the batches whose hidden state contributed to the final prediction. In practice, variable length backpropagation sequences are used (Merity et al., 2017a).

Bidirectional language model
Both, a forward and backward language model are pre-trained. The classifier is then fine-tuned for both the language models independently. The average of the two classifier predictions is taken as the final output.

Further Reading

• The research paper – Understanding the Working of Universal Language Model Fine Tuning (ULMFiT)
• Cutting Edge Deep Learning for coders –  Lesson 10, Part2 2018, fast.ai

• Introducing state of the art text classification with universal language models

• Pre-trained Wikitext 103 model and vocab

(Edit) A big thanks to Jeremy Howard for the shout-out

Let’s go through how a typical pivot operation looks like –

        Pclass   Survived  Sex
0	3	 0	   male
1	1	 1	   female
2	3	 1	   female
3	1	 1	   female
4	3	 0	   male
5	3	 0	   male
6	1	 0	   male
7	3	 0	   male
8	3	 1	   female
9	2	 1	   male

You have an index column – Pclass, a values column – Survived, and the column on which you need to perform the pivot – Sex. After the pivot operation, the resulting table would look like –

	Pclass	female_survived	male_survived
0	1	2	        0
1	2	1	        1
2	3	2	        0

In the above example, we can get the desired pivoted table using the following query –

select Pclass, 
sum(case when Sex = "male" then Survived else 0 end) as male_survived,
sum(case when Sex = "female" then Survived else 0 end) as female_survived
from `table_name`
group by 1

Writing case statements for each unique value in the pivot column works well if the number of unique values is small. But, imagine writing these queries for a column that has hundreds of unique values. It seems that we will have to either –

• Write these annoying case statements again and again for each unique value of the column.
•  Or switch to Python or R and use a package like pandas to pivot the table. But then we would lose the “BiqQuery advantage” of almost real-time analysis if the table is too large.

After some initial struggle, I found a middle ground. As you can notice the case statements are mostly redundant. If we can get a list of all the unique values, we can simply loop over the list to generate a SQL query. Hence, I started writing loops in Python to generate the repetitive case statements which I then used to run in BiqQuery.

But soon I realized there is still some room for automation. Hence, I came up with the Python module BqPivot. It takes the names of the various columns and the data as input and outputs a SQL query that can then be used in BigQuery to pivot your table. Let me give you a step-by-step introduction –

In order to run this, you need to have Python 3 and pandas installed on your system. Next, you need to copy this file in your working directory. You can find it here. After this, you need to import the BqPivot class.

from bq_pivot import BqPivot

Then you need to create an object of this class by passing the required arguments.

import pandas as pd

# The following example uses the kaggle's titanic data. It can be found here -
# `https://www.kaggle.com/c/titanic/data`
data = pd.read_csv("titanic.csv")

# Using only the top 20 rows for demonstration -
data = data.head(20)

gen = BqPivot(data=data,
              index_col=["Pclass", "Sex"],
              pivot_col="Cabin",
              values_col="Survived",
              agg_fun="sum")

Let’s go through the input arguments –

• data (the string path to the CSV file or a pandas data frame object) – The required data is quite flexible. It can work with the original table from BigQuery. But if that table is huge and you do not want to load the entire data into your machine you can also only provide the unique values of the pivot column in your data. Therefore, a data generated from a group by on the pivot column would also work perfectly fine.

select Cabin from `titanic_data` group by 1

• index_col (a Python list or string) – The names of the index columns in the query on which the group by needs to be performed. Again, these columns may or may not be present in the input data. The only requirement is that these names must be consistent with the original data on which the output query would be run.
• pivot_col (string) – The name of the column on which the pivot needs to be performed. This column must be present in the input data.
• values_col (string) – The name of the column on which aggregation needs to be performed. Just like the index_col the name should only be consistent with the original data and may not be present in the data argument.

Let’s also go through the optional arguments –

• agg_fun (string, optional) – The name of the SQL aggregation function. It defaults to the sum function. Examples – “avg”, “min”, “count” etc.
• table_name (string, optional) – The name of the table in the query. If not provided it is replaced by a placeholder which can be changed later by the actual table name manually.
• not_eq_default (numeric, optional) The value to take when the else statement is satisfied. For example, if one is doing a sum aggregation on the value column then the not_eq_default should be equal to 0. Because the case statement part of the SQL query would look like –

sum(case when Cabin = C85 then Survived else 0)

Similarly, if the aggregation function is min then the not_eq_default should be positive infinity. This argument defaults to 0.

• add_col_nm_suffix (boolean, optional) – If True, then the original values column name will be added as a suffix in the new pivoted columns. It defaults to True.
• custom_agg_fun (string, optional) – This can be used if one wants to give customized aggregation function. The values column name should be replaced with {}. For example, if we want an aggregation function like – sum(coalesce(Survived, 0)) then the custom_agg_fun argument would be – sum(coalesce({}, 0)). It defaults to none. If provided it overrides the agg_fun parameter.
• prefix (string, optional) – A fixed string to add as a prefix in the pivoted column names. It defaults to None.
• suffix (string, optional) – A fixed string to add as a suffix in the pivoted column names. It defaults to None.

Now that we have covered all the possible inputs let’s run the command to generate the SQL query. We can run the write_query method to save the generated query in a text file.

gen.write_query("query.txt")

On opening the query.txt file you can find the following query –

select Pclass, Sex, 
sum(case when Cabin = "nan" then Survived else 0 end) as nan_survived,
sum(case when Cabin = "C85" then Survived else 0 end) as c85_survived,
sum(case when Cabin = "C123" then Survived else 0 end) as c123_survived,
sum(case when Cabin = "E46" then Survived else 0 end) as e46_survived,
sum(case when Cabin = "G6" then Survived else 0 end) as g6_survived,
sum(case when Cabin = "C103" then Survived else 0 end) as c103_survived
from <--insert-table-name-here-->
group by 1,2

I hope this will help you save some time while pivoting tables in BigQuery. If you have any questions, let me know in the comments section. I will do my best to answer. Before I sign out, here are some pointers on when and when not to use this module.

When to use this module?

• When you want a lot of pivot columns and you do not have an intern who would write the query for you.
• When your data set is very large and pivoting in Python or R would not give you real-time results.

When not to use this module?

• When the number of unique values in the pivot column is less than 10.
• When the number of unique values in the pivot column is more than 10,000. This would result in more than 10,000 columns in the resulting table. But BigQuery only supports tables with less than 10,000 columns. 
• When you are willing to work hard but not smart.

Thank You.

In this blog post, I discuss 6 things I wish I knew about Python as a beginner. Using these in your code would not only make your code more beautiful but also more efficient. All these tips are applicable for both Python 2 and 3.

1. Looping Over Collection and Indices

If you are coming from a C++ or a Java background, your logic for looping over collections and indices may look like –

>>> characters = ["Rick", "Maggie", "Carl", "Negan", "Daryl"]

>>> for i in range(len(characters)):
       print(i, characters[i])
(0, 'Rick')
(1, 'Maggie')
(2, 'Carl')
(3, 'Negan')
(4, 'Daryl')

But Python has a much cleaner way to do it. You can use the enumerate function. It is fast, beautiful and also saves us from tracking the indices.

>>> for i, character in enumerate(characters):
       print(i, character)
(0, 'Rick')
(1, 'Maggie')
(2, 'Carl')
(3, 'Negan')
(4, 'Daryl')

Another useful way to use enumerate is when we need a dictionary with keys as indices of the list and values as elements of the list.

>>> dict(enumerate(characters))
{0: 'Rick', 1: 'Maggie', 2: 'Carl', 3: 'Negan', 4: 'Daryl'}

2. Looping over two collections

Looping over two lists (or other collections) together is also way simpler than you might think. So, instead of –

>>> characters = ["Tyrion", "Arya", "Danerys", "John", "Jeoffery"]
>>> houses = ["Lannister", "Stark", "Targaryen", "Targaryen"]

>>> length = min(len(characters), len(houses))
>>> for i in range(length):
	print(characters[i], houses[i])

('Tyrion', 'Lannister')
('Arya', 'Stark')
('Danerys', 'Targaryen')
('John', 'Targaryen')

You can simply use the zip method –

>>> for character, house in zip(characters, houses):
	print(character, house)

('Tyrion', 'Lannister')
('Arya', 'Stark')
('Danerys', 'Targaryen')
('John', 'Targaryen')

Similar to enumerate the zip function can also be used to create dictionaries directly,

>>> dict(zip(characters, houses))
{'Arya': 'Stark', 'John': 'Targaryen', 'Danerys': 'Targaryen',
 'Tyrion': 'Lannister'}

Note – The zip function behaves differently in Python 2 and 3. In Python 2 it creates a new third list. Thus increasing memory requirement. Therefore, a better alternative to zip in Python 2 is itertools.izip. It behaves exactly as the zip method described above but is scalable as well.

In Python 3, this problem was fixed. Similar to the itertools.izip method the zip function also returns a generator, thus making it memory efficient.

3. Concatenating Strings

I see many people over complicate string concatenation and ignore the join method of strings. So instead of doing –

>>> s = ''
>>> sentence = ["I", "love", "to", "code", "in", "Python"]
>>> for word in sentence:
	s += word + " "	
>>> s
'I love to code in Python '

We can simply do –

>>> sentence = ["I", "love", "to", "code", "in", "Python"]

>>> " ".join(sentence)
'I love to code in Python'

>>> "".join(sentence)
'IlovetocodeinPython'

>>> "|".join(sentence)
'I|love|to|code|in|Python'

4. Counting Using Dictionaries

How many of us have written something similar to –

>>> memo = {}
>>> sentence = "I love to code in Python."
>>> for char in sentence:
        if char in memo:
	    memo[char] += 1
	else:
	    memo[char] = 1

Here, we needed to calculate the occurrence of each character. I had also been using the above code until I was introduced to the defaultdict collection which lets us do all this in a more concise manner.

>>> from collections import defaultdict
>>> memo = defaultdict(int)
>>> for char in sentence:
	memo[char] += 1

A Python dictionary throws a KeyError if we try to get an item with a key that is not present in the dictionary. Unlike a dictionary, when we try to access an item whose key is not already present then the defaultdict will create an item on its own. This created item depends on the function we passed while declaring the defaultdict object. Here, the object is created using int() function that will return the integer object 0.

Similarly, a defaultdict object created using the list() function would return an empty list for a key that is not already present.

>>> s = [('2017', 1), ('2018', 2), ('2017', 3), ('2019', 4), ('2019', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
        d[k].append(v)

>>> d.items()
[('2017', [1, 3]), ('2019', [4, 1]), ('2018', [2])]

5. List and Generator Comprehensions

Comprehensions is a powerful syntax to transform collections in various ways. This is perhaps my favorite feature in Python. Let’s explore this with examples.

Suppose, you have to create a list of squares of the first 10 natural numbers (1, 4, 9, …..). Without using list comprehension your code will look messy. Let me show you.

>>> elems = []
>>> for x in range(1, 11):
	elems.append(x*x)

>>> elems
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Or how about if we only print the square of even numbers and leave out the odd numbers.

>>> elems = []
>>> for x in range(1, 11):
	if(x % 2 == 0):
	    elems.append(x*x)
>>> elems
[4, 16, 36, 64, 100]

Let’s take this one step further. Let’s print the square of even numbers and cube of odd numbers.

>>> for x in range(1, 11):
	if(x % 2 == 0):
	    elems.append(x*x)
	else:
	    elems.append(x*x*x)	
>>> elems
[1, 4, 27, 16, 125, 36, 343, 64, 729, 100]

Even if this does not seem ugly enough for you, let me show you the comprehension syntax to do it and I bet you will love it.

>>> [x*x for x in range(1, 11)]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

>>> [x*x for x in range(1, 11) if x % 2 == 0]
[4, 16, 36, 64, 100]

>>> [x*x if x % 2 == 0 else x*x*x for x in range(1, 11)]
[1, 4, 27, 16, 125, 36, 343, 64, 729, 100]

The comprehension syntax is as useful and beautiful with dictionaries as it is for lists. Let’s create a dictionary with keys as strings and values as the length of strings.

>>> {x: len(x) for x in ["Ted", "Barney", "Robin", "Marshall", "Lily"]}
{'Lily': 4, 'Marshall': 8, 'Barney': 6, 'Ted': 3, 'Robin': 5}

Comprehensions can also be used for assignment.

>>> x = 123
>>> number = "even" if x % 2 == 0 else "odd"
>>> number
'odd'

6. Tuple Assignment

Tuple assignments are very clean in Python unless you try to mess them up. So next time when you find something like this in your code –

>>> movies = ("The Dark Knight", "Inception", "Interstellar", "Prestige")
>>> m1 = movies[0]
>>> m2 = movies[1]
>>> m3 = movies[2]
>>> m4 = movies[3]
>>> m1, m2, m3, m4
('The Dark Knight', 'Inception', 'Interstellar', 'Prestige')

You can replace it with –

>>> m1, m2, m3, m4 = movies
>>> m1, m2, m3, m4
('The Dark Knight', 'Inception', 'Interstellar', 'Prestige')

Or something like this –

>>> x = 1
>>> y = 2
>>> tmp = x
>>> x = x + y
>>> y = tmp
>>> x, y
(3, 1)

with –

>>> x, y = 1, 2
>>> x, y = x + y, x
>>> x, y
(3, 1)

Hope you enjoyed reading this. Thank a lot.