IMDB Movie Classification Grid Search

Hyperparameter tuning for DNNs tends to be a bit more involved than other ML models due to the number of hyperparameters that can/should be assessed and the dependencies between these parameters. To automate the hyperparameter tuning for keras and tensorflow, we use the tfruns package.

This notebook shows an example of performing a grid search on a densley connected feedforward neural network for the IMDB movie review classifier introduced in this case study.

library(tfruns)
library(dplyr)

tfruns provides added flexibility for tracking, visualizing, and managing training runs. The most common way to use tfruns is to create an R script that contains the code to be executed. For this example, I created the imdb-grid-search.R script.

Within this script, we create “flags”, which identify the hyperparameters of interest. For this example, I assess:

• batch sizes of 128 and 512
• layers of 1, 2, and 3
• number of units per hidden layer of 16, 32, and 64
• learning rate of 0.001 and 0.0001
• dropout rates of 0, 0.3, and 0.5
• weight decay of 0, 0.01, and 0.001

This equates to 324 models. Since these models run relatively quickly, and since I ran this at the end of the day, I performed a full cartesian grid search. This means I ran and assessed every single one of the 324 models. If you are pressed for time you can run a stochastic hyperparameter grid search by using the sample parameter in tuning_run below.

Next, to execute this grid search, you first specify the grid search hyperparameter values for the flags you created in your .R script like below. You then run tuning_run() in place of source() to execute your .R script for the supplied hyperparameter grid.

Note: this takes over 3 hours to run on a non-GPU

grid_search <- list(
  batch_size = c(128, 512),
  layers = c(1, 2, 3),
  units = c(16, 32, 64),
  learning_rate = c(0.001, 0.0001),
  dropout = c(0, 0.3, 0.5),
  weight_decay = c(0, 0.01, 0.001)
)

tuning_run("imdb-grid-search.R", flags = grid_search, confirm = FALSE, echo = FALSE)

This grid search execution will create a “runs” subdirectory within your working directory. This fold contains information for every single training run executed during the grid search.

To list the results you can run ls_runs():

data.frame(ls_runs(runs_dir = "imdb_runs"))

You can even filter and order the results. The following illustrates that a few models tied with the lowest loss score of 0.262.

ls_runs(runs_dir = "imdb_runs", order = eval_best_loss, decreasing = FALSE)

Data frame: 324 x 31 

# ... with 314 more rows
# ... with 23 more columns:
#   flag_batch_size, flag_layers, flag_units, flag_learning_rate, flag_dropout,
#   flag_weight_decay, samples, batch_size, epochs, epochs_completed, metrics, model,
#   loss_function, optimizer, learning_rate, script, start, end, completed, output,
#   source_code, context, type

To see details about any one of these models you can run view_run(). In this example, I take the first optimal model from above. When you execute this, a pop up window will appear with that models summary information as illustrated below.

best_run <- ls_runs(
  runs_dir = "imdb_runs",
  order = eval_best_loss,
  decreasing = FALSE
  ) %>%
  slice(1) %>%
  pull(run_dir)

view_run(best_run)

There are many other handy features to the tfruns package. I suggest you check it out at https://tensorflow.rstudio.com/tools/tfruns/overview/ and take it for a test drive.