Grid Search for Amazon Review Helpfulness with Word Embeddings

This notebook shows an example of performing a grid search for the Amazon review helpfulness project. If you are unfamiliar with Keras/Tensorflow grid searches please review this notebook first.

library(tfruns)
library(dplyr)

Our grid search setup is located in this amazon-embeddings-grid-search.R script. The grid we’re going to search across is below and contains 5,832 hyperparameter combinations. This is way too large to evaluate all possible combinations so, rather, we’ll perform a random search which typically finds a near optimal solution. We execute a random search by including a sample argument less than 1.

Here, we evaluate only 10% (583 models) of the possible hyperparameter combinations. In fact, I even cut the grid search short which resulted in 137 models being evaluated.

Note: this will take several hours to run on a non-GPU

grid_search <- list(
  top_n_words = c(5000, 10000, 20000),
  max_len = c(75, 150, 200),
  output_dim = c(16, 32, 64),
  learning_rate = c(0.001, 0.0001),
  batch_size = c(32, 64, 128),
  layers = c(1, 2, 3),
  units = c(16, 32, 128),
  dropout = c(0, 0.5),
  weight_decay = c(0, 0.01)
)

tuning_run(
  "amazon-embeddings-grid-search.R",
  flags = grid_search,
  runs_dir = "amazon_runs",
  sample = 0.1,
  confirm = FALSE,
  echo = FALSE
  )

This grid search execution will create an “amazon_runs” subdirectory within your working directory. Our results indicate that the best performing models all have optimal MSEs in the low 0.05 range.

ls_runs(runs_dir = "amazon_runs", order = eval_best_loss, decreasing = FALSE)

Data frame: 137 x 38 

# ... with 127 more rows
# ... with 27 more columns:
#   flag_top_n_words, flag_max_len, flag_output_dim, flag_learning_rate, flag_batch_size, flag_layers, flag_units,
#   flag_dropout, flag_weight_decay, samples, batch_size, steps_completed, epochs, epochs_completed, metrics, model,
#   loss_function, optimizer, learning_rate, script, start, end, completed, output, source_code, context, type

If we look at our optimal model we see it has the following parameters:

• top_n_words: 20,000
• max_len: 200
• output_dim: 32
• layers (hidden layers in classifier): 1
• units (in hidden layers): 32
• learning_rate: 0.0001
• batch_size: 128
• dropout: 0.5
• weight_decay: 0

best_run <- ls_runs(
  runs_dir = "amazon_runs",
  order = eval_best_loss,
  decreasing = FALSE
  ) %>%
  slice(1) %>%
  pull(run_dir)

view_run(best_run)