Introduction to RNNs & LSTMs

In this example, we are going to learn about recurrent neural networks (RNNs) and long short term memory (LSTM) neural networks. These architectures are designed for sequence data, which can include text, videos, time series, and more. However, for this workshop we are going to focus on text data as this is the domain seeing the largest impact from LSTMs.

Learning objectives:

• Understand how an RNN works, how to implement it, and its primary weakness
• Understand how an LSTM works and how to implement it

Requirements

library(keras)
library(tidyverse)
library(glue)

Prepare our data

For our introduction we’re going to start with the built-in IMDB data set. Let’s use the 10,000 most frequent words and we will set the max length of each review to be 500, which will capture the majority of most reviews.

# establish our data characteristics
n_features <- 10000
max_len <- 500

# import and prep our data
imdb <- dataset_imdb(num_words = n_features)
c(c(x_train, y_train), c(x_test, y_test)) %<-% imdb

x_train <- pad_sequences(x_train, maxlen = max_len)
x_test <- pad_sequences(x_test, maxlen = max_len)

RNNs

To model our IMDB data, we could use:

1. one-hot encoding, which has a problem with only allowing a single representation of a word. “Seattle” is simply a word but has no relationship to “Seahawks”, “Wilson”, “weather”, or “Starbucks”.
2. embeddings, which allow our inputs (i.e. words) to have relationships with other inputs but does not control for the sequential nature of the inputs.

Recurrent neural networks were specifically designed to model sequential data. ℹ️

Train an RNN

Let’s build a model with an RNN layer. First, we still use an embedding layer to allow for more complex representation of our words but we follow that with layer_simple_rnn(). Everything else remains the same as before.

model <- keras_model_sequential() %>%
  layer_embedding(
    input_dim = n_features,
    input_length = max_len,
    output_dim = 32, 
    name = "Embeddings") %>%
  layer_simple_rnn(units = 32, name = "RNN") %>%
  layer_dense(units = 1, activation = "sigmoid", name = "Prediction")

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = "accuracy"
  )

summary(model)

Model: "sequential_1"
________________________________________________________________________________________
Layer (type)                           Output Shape                       Param #       
========================================================================================
Embeddings (Embedding)                 (None, 500, 32)                    320000        
________________________________________________________________________________________
RNN (SimpleRNN)                        (None, 32)                         2080          
________________________________________________________________________________________
Prediction (Dense)                     (None, 1)                          33            
========================================================================================
Total params: 322,113
Trainable params: 322,113
Non-trainable params: 0
________________________________________________________________________________________

history <- model %>% fit(
  x_train, y_train,
  epochs = 10,
  batch_size = 128,
  validation_split = 0.2,
  callbacks = list(callback_early_stopping(patience = 3))
)

best_epoch <- which(history$metrics$val_loss == min(history$metrics$val_loss))
loss <- history$metrics$val_loss[best_epoch] %>% round(3)
acc <- history$metrics$val_acc[best_epoch] %>% round(3)

glue("The best epoch had a loss of {loss} and an accuracy of {acc}")

The best epoch had a loss of 0.36 and an accuracy of 0.852

Your Turn! (5min)

Spend a few minutes adjusting this model and see how it impacts performance. You may want to test:

• Does increasing and decreasing the word embedding dimension and/or the number of RNN layer units impact performance?
• It’s sometimes userful to stack several recurrent layers one after the other in order to increase the representational power of a network. What happens when you try this?

Tip: An RNN will take inputs from different parts of a sequence and produce a single output. When stacking multiple recurrent layers, you need to set return_sequences = TRUE so that you get the output for each part of the sequence so that you can supply a sequence into the next RNN layer (see ?layer_simple_rnn()).

model <- keras_model_sequential() %>%
  layer_embedding(
    input_dim = n_features,
    input_length = max_len,
    output_dim = 32, 
    name = "Embeddings") %>%
  layer_simple_rnn(units = 32, name = "RNN_1", return_sequences = TRUE) %>%
  layer_simple_rnn(units = 32, name = "RNN_2") %>%
  layer_dense(units = 1, activation = "sigmoid", name = "Prediction")

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = "accuracy"
  )

summary(model)

history <- model %>% fit(
  x_train, y_train,
  epochs = 10,
  batch_size = 128,
  validation_split = 0.2,
  callbacks = list(callback_early_stopping(patience = _____))
)

best_epoch <- which(history$metrics$val_loss == min(history$metrics$val_loss))
loss <- history$metrics$val_loss[best_epoch] %>% round(3)
acc <- history$metrics$val_acc[best_epoch] %>% round(3)

glue("The best epoch had a loss of {loss} and an accuracy of {acc}")

LSTMs

Unfortunately, a problem with RNNs (and any very deep neural network) is as our network gets deep, it loses the signal due to the vanishing gradient descent problem. LSTMs were developed to address this problem. ℹ️

Train an LSTM

To train an LSTM, we simply replace layer_simple_rnn() with layer_lstm().

model <- keras_model_sequential() %>%
  layer_embedding(
    input_dim = n_features,
    input_length = max_len,
    output_dim = 32, 
    name = "Embeddings") %>%
  layer_lstm(units = 32, name = "LSTM") %>%
  layer_dense(units = 1, activation = "sigmoid", name = "Prediction")

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = "accuracy"
  )

summary(model)

Model: "sequential_2"
________________________________________________________________________________________
Layer (type)                           Output Shape                       Param #       
========================================================================================
Embeddings (Embedding)                 (None, 500, 32)                    320000        
________________________________________________________________________________________
LSTM (LSTM)                            (None, 32)                         8320          
________________________________________________________________________________________
Prediction (Dense)                     (None, 1)                          33            
========================================================================================
Total params: 328,353
Trainable params: 328,353
Non-trainable params: 0
________________________________________________________________________________________

history <- model %>% fit(
  x_train, y_train,
  epochs = 10,
  batch_size = 128,
  validation_split = 0.2,
  callbacks = list(callback_early_stopping(patience = 3))
)

best_epoch <- which(history$metrics$val_loss == min(history$metrics$val_loss))
loss <- history$metrics$val_loss[best_epoch] %>% round(3)
acc <- history$metrics$val_acc[best_epoch] %>% round(3)

glue("The best epoch had a loss of {loss} and an accuracy of {acc}")

The best epoch had a loss of 0.31 and an accuracy of 0.885

Your Turn! (5min)

Spend a few minutes adjusting this model and see how it impacts performance. You may want to test:

• Does increasing and decreasing the word embedding dimension and/or the number of LSTM layer units impact performance?
• As we learned in an earlier module, dropout can help reduce overfitting and improve model performance. See if you can add dropout to your model. Hint, checkout the docs (?layer_lstm()) because adding dropout to LSTMs are a little unique.

Tip: When applying dropout to a recurrent network, it’s important to apply the same dropout pattern to each timestep within the recurrent layer, rather than randomly. Consequently, recurrent layers have two dropout parameters:

• dropout: the dropout rate for inputs going into the recurrent layer
• recurrent_dropout: the dropout rate for the recurrent units within the recurrent layer

See ?layer_lstm() for details. Note, you can still apply dropout between the embedding layer and the dense layer after the LSTM layer by using the regular layer_dropout() layer.

model <- keras_model_sequential() %>%
  layer_embedding(
    input_dim = n_features,
    input_length = max_len,
    output_dim = 32, 
    name = "Embeddings") %>%
  layer_lstm(units = 32, dropout = 0.3, recurrent_dropout = 0.3, name = "LSTM") %>%
  layer_dense(units = 1, activation = "sigmoid", name = "Prediction")

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = "accuracy"
  )

summary(model)

history <- model %>% fit(
  x_train, y_train,
  epochs = 10,
  batch_size = 128,
  validation_split = 0.2,
  callbacks = _____
)

best_epoch <- which(history$metrics$val_loss == min(history$metrics$val_loss))
loss <- history$metrics$val_loss[best_epoch] %>% round(3)
acc <- history$metrics$val_acc[best_epoch] %>% round(3)

glue("The best epoch had a loss of {loss} and an accuracy of {acc}")

Why the lack of accuracy

So why are our recurrent models not doing much better than our one-hot encoding and simple word embedding models? For most sentiment predictions we can capture the essence of the tone (i.e. positive vs. negative) by focusing on a few key words, which our simpler encoding models can do quite well.

As your prediction task requires more spatial context, then recurrent networks should begin to out perform the more simpler encoding models.

Takeaways

• Recurrent networks help to capture spatial context within text
• RNNs
  • are just multiple perceptrons sequentially linked
  • pass information through hidden states
  • use the Tanh activation to control value magnitudes
  • suffer from the vanishing gradient
  • implemented with layer_simple_rnn()
• LSTMs
  • use multiple gates that help to “remember” or “forget” information
  • computationally expensive
  • implemented with layer_lstm()
• Unique characteristics
  • must use return_sequences = TRUE to stack multiple recurrent layers
  • recurrent layers have their own special dropout procedures –> don’t use layer_dropout(); rather, use the dropout parameters within the recurrent layer.

🏠