Applied Machine Learning

class: title-slide, center

# Applied Machine Learning

## Resampling and Grid Search

---
# Loading

```r
library(tidymodels)
```

```
## ── Attaching packages ─────────────────────────────────────────────────────────── tidymodels 0.0.4 ──
```

```
## ✓ broom     0.5.3     ✓ recipes   0.1.9
## ✓ dials     0.0.4     ✓ rsample   0.0.5
## ✓ dplyr     0.8.3     ✓ tibble    2.1.3
## ✓ ggplot2   3.2.1     ✓ tune      0.0.1
## ✓ infer     0.5.1     ✓ workflows 0.1.0
## ✓ parsnip   0.0.5     ✓ yardstick 0.0.5
## ✓ purrr     0.3.3
```

```
## ── Conflicts ────────────────────────────────────────────────────────────── tidymodels_conflicts() ──
## x purrr::discard()    masks scales::discard()
## x dplyr::filter()     masks stats::filter()
## x dplyr::lag()        masks stats::lag()
## x ggplot2::margin()   masks dials::margin()
## x recipes::step()     masks stats::step()
## x recipes::yj_trans() masks scales::yj_trans()
```

---

# Previously

```r
library(AmesHousing)

ames <- make_ames() %>% 
  dplyr::select(-matches("Qu"))

set.seed(4595)

data_split <- initial_split(ames, strata = "Sale_Price")
ames_train <- training(data_split)
ames_test  <- testing(data_split)

perf_metrics <- metric_set(rmse, rsq, ccc)
```

```r
ames_rec <- recipe(
    Sale_Price ~ Bldg_Type + Neighborhood + Year_Built + 
                 Gr_Liv_Area + Full_Bath + Year_Sold + Lot_Area +
                 Central_Air + Longitude + Latitude,
    data = ames_train
  ) %>%
  step_log(Sale_Price, base = 10) %>%
  step_BoxCox(Lot_Area, Gr_Liv_Area) %>%
  step_other(Neighborhood, threshold = 0.05)  %>%
  step_dummy(all_nominal()) %>%
  step_interact(~ starts_with("Central_Air"):Year_Built) %>%
  step_bs(Longitude, Latitude, deg_free = 5)
```

---
layout: false
class: inverse, middle, center

#  Resampling

---

# Resampling Methods

.pull-left[
These are additional data splitting schemes that are applied to the _training_ set and are used for **estimating model performance**.

They attempt to simulate slightly different versions of the training set. These versions of the original are split into two model subsets:

* The _analysis set_ is used to fit the model (analogous to the training set). 
* Performance is determined using the _assessment set_.

This process is repeated many times.

]

.pull-right[

]

There are [different flavors of resampling](https://bookdown.org/max/FES/resampling.html) but we will focus on one method in these notes.

---

# The Model Workflow and Resampling

All resampling methods repeat this process multiple times:

<br>

<br>

The final resampling estimate is the average of all of the estimated metrics (e.g. RMSE, etc).

---

# V-Fold Cross-Validation

.pull-left[

Here, we randomly split the training data into _V_ distinct blocks of roughly equal size (AKA the "folds").

* We leave out the first block of analysis data and fit a model.

* This model is used to predict the held-out block of assessment data.

* We continue this process until we've predicted all _V_ assessment blocks

The final performance is based on the hold-out predictions by _averaging_ the statistics from the _V_ blocks.

]

.pull-right[

<br>

_V_ is usually taken to be 5 or 10 and leave-one-out cross-validation has each sample as a block.

**Repeated CV** can be used when training set sizes are small. 5 repeats of 10-fold CV averages for 50 sets of metrics.

]

---

#  10-Fold Cross-Validation with _n_ = 50

---

# Resampling Results

The goal of resampling is to produce a single estimate of performance for a model.

Even though we end up estimating _V_ models (for _V_-fold CV), these models are discarded after we have our performance estimate.

Resampling is basically an _emprical simulation system_ used to understand how well the model would work on _new data_.

---

# Cross-Validating Using {rsample} <img src="images/rsample.png" class="title-hex">

rsample has a number of resampling functions built in. One is `vfold_cv()`, for performing V-Fold cross-validation like we've been discussing.

```r
set.seed(2453)

cv_splits <- vfold_cv(ames_train) #10-fold is default

cv_splits
```

```
## #  10-fold cross-validation 
## # A tibble: 10 x 2
##    splits           id    
##    <named list>     <chr> 
##  1 <split [2K/220]> Fold01
##  2 <split [2K/220]> Fold02
##  3 <split [2K/220]> Fold03
##  4 <split [2K/220]> Fold04
##  5 <split [2K/220]> Fold05
##  6 <split [2K/220]> Fold06
##  7 <split [2K/220]> Fold07
##  8 <split [2K/220]> Fold08
##  9 <split [2K/220]> Fold09
## 10 <split [2K/219]> Fold10
```

???

Note that `<split [2K/222]>` rounds to the thousandth and is the same as `<1977/222/2199>`

---

# Cross-Validating Using {rsample} <img src="images/rsample.png" class="title-hex">

- Each individual split object is similar to the `initial_split()` example.

- Use `analysis()` to extract the resample's data used for the fitting process.

- Use `assessment()` to extract the resample's data used for the performance process.

.pull-left[

```r
cv_splits$splits[[1]]
```

```
## <1979/220/2199>
```

]

.pull-right[

```r
cv_splits$splits[[1]] %>% 
  analysis() %>%
  dim()
```

```
## [1] 1979   74
```

```r
cv_splits$splits[[1]] %>% 
  assessment() %>%
  dim()
```

```
## [1] 220  74
```

]

---

# K-Nearest Neighbors Model

_K_-nearest neighbors stores the training set (including the outcome).

When a new sample is predicted, _K_ training set points are found that are most similar to the new sample being predicted.

The predicted value for the new sample is some summary statistic of the neighbors, usually:

* the mean for regression, or
* the mode for classification.

Let's try a 5-neighbor model on the Ames data.

---

# _5_-Nearest Neighbors Model

<div id="htmlwidget-d8037c81783ee8105465" style="width:80%;height:504px;" class="leaflet html-widget"></div>
<script type="application/json" data-for="htmlwidget-d8037c81783ee8105465">{"x":{"options":{"crs":{"crsClass":"L.CRS.EPSG3857","code":null,"proj4def":null,"projectedBounds":null,"options":{}}},"setView":[[42.060936,-93.641325],18,[]],"calls":[{"method":"addProviderTiles","args":["CartoDB.DarkMatterNoLabels",null,null,{"errorTileUrl":"","noWrap":false,"detectRetina":false}]},{"method":"addCircles","args":[[42.060977,42.061114,42.061149,42.061318,42.061193],[-93.641487,-93.641488,-93.641772,-93.641781,-93.642447],6,null,null,{"interactive":true,"className":"","stroke":true,"color":"red","weight":5,"opacity":0.4,"fill":true,"fillColor":"red","fillOpacity":0.2},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null,null]},{"method":"addCircles","args":[42.060936,-93.641325,6,null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":5,"opacity":0.4,"fill":true,"fillColor":"yellow","fillOpacity":0.2},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null,null]},{"method":"addCircles","args":[[42.052659,42.051245,42.060899,42.060779,42.062978,42.060728,42.06112,42.059193,42.05848,42.058151,42.057268,42.059169,42.061239,42.0603514,42.056673,42.054453,42.05027,42.049297,42.054592,42.055227,42.053395,42.056375,42.055318,42.051835,42.051685,42.049672,42.050683,42.050681,42.049811,42.062352,42.062109,42.0611331,42.060872,42.060879,42.059617,42.058599,42.057102,42.057164,42.058252,42.057181,42.061928,42.06257,42.062303,42.054386,42.053243,42.051721,42.051674,42.052437,42.050909,42.05306,42.051653,42.050084,42.051429,42.036113,42.036026,42.036659,42.034589,42.0347,42.036633,42.034555,42.034678,42.036535,42.034793,42.034856,42.03457,42.034603,42.046825,42.045773,42.047452,42.048086,42.048566,42.047899,42.047876,42.047782,42.0471748,42.0470316,42.04638,42.043704,42.034561,42.048606,42.046322,42.043306,42.043329,42.043822,42.043821,42.046574,42.042205,42.043363,42.039085,42.038971,42.040391,42.0418387,42.039211,42.039211,42.038538,42.035256,42.035482,42.035943,42.036095,42.037424,42.047447,42.043984,42.042908,42.0430839,42.043969,42.044052,42.0435919,42.042308,42.040396,42.04137,42.038515,42.039841,42.038443,42.038521,42.0381282,42.035869,42.0372097,42.040085,42.041119,42.035867,42.03594,42.0347571,42.034848,42.034748,42.034851,42.036873,42.037093,42.033313,42.031363,42.031381,42.0303783,42.033644,42.03128,42.0303113,42.030383,42.030467,42.030388,42.030331,42.029146,42.028145,42.028164,42.02916,42.028165,42.033642,42.033621,42.0336,42.033665,42.033631,42.032302,42.027986,42.028888,42.028875,42.026914,42.020693,42.024141,42.026741,42.033454,42.028256,42.026752,42.02616,42.025255,42.034451,42.033386,42.033387,42.033008,42.03237,42.029867,42.03069,42.029924,42.029731,42.030519,42.03136,42.03128,42.030082,42.03009,42.03387,42.026547,42.024673,42.023079,42.028226,42.026533,42.024208,42.025173,42.0247497,42.025019,42.024162,42.032076,42.033783,42.03068,42.025747,42.024604,42.024604,42.030601,42.0298532,42.024318,42.021321,42.01867,42.022186,42.022045,42.021395,42.021133,42.018662,42.019069,42.016121,42.018721,42.017749,42.016275,42.016253,42.01404,42.014023,42.0203585,42.021245,42.020635,42.021163,42.020306,42.019849,42.019629,42.020196,42.021793,42.022674,42.022702,42.0177987,42.019091,42.019093,42.014937,42.014682,42.011005,42.010606,42.009337,42.010797,42.022772,42.021414,42.021474,42.021456,42.021327,42.02152,42.008687,41.99706,41.997052,41.996065,41.994013,41.997741,41.995346,41.994284,41.99353,41.995489,41.99435,41.992989,41.99321,41.990934,41.991861,41.991758,41.991876,41.992212,41.9917,41.991824,41.991558,41.988729,41.9889358,41.986647,41.9913,41.989848,41.989185,42.05139,42.059956,42.060896,42.060598,42.061178,42.058274,42.058059,42.059645,42.058194,42.05793,42.058173,42.057854,42.058747,42.059242,42.059817,42.058439,42.055408,42.05184,42.052507,42.052102,42.052719,42.05288,42.050345,42.049394,42.051619,42.0503771,42.049406,42.053166,42.052448,42.05267,42.04891,42.056398,42.054626,42.055184,42.05337,42.053306,42.053307,42.0564419,42.056461,42.052758,42.052338,42.051661,42.051823,42.051822,42.050705,42.050682,42.050533,42.050213,42.05009,42.050134,42.050222,42.0488934,42.048486,42.0623099,42.063304,42.063025,42.061902,42.062991,42.062992,42.062994,42.062942,42.062258,42.062776,42.062789,42.0623985,42.062108,42.060934,42.060353,42.061382,42.061437,42.060576,42.061678,42.060178,42.0599123,42.0598989,42.061404,42.059241,42.058886,42.0592,42.059177,42.059179,42.058336,42.057177,42.057089,42.057019,42.056983,42.057467,42.06232,42.062562,42.062305,42.062081,42.062086,42.061562,42.059377,42.061739,42.061656,42.061365,42.063283,42.058368,42.05859,42.059481,42.059068,42.057839,42.057834,42.057833,42.057093,42.056955,42.058668,42.053749,42.053911,42.055626,42.053441,42.053041,42.052668,42.050411,42.049701,42.05161,42.050215,42.049921,42.050971,42.054282,42.055036,42.054302,42.054152,42.053258,42.053108,42.051797,42.051645,42.051648,42.051652,42.050406,42.052346,42.052127,42.05142,42.051378,42.051251,42.037611,42.037661,42.036011,42.035282,42.038127,42.037756,42.037434,42.036852,42.036905,42.036576,42.035518,42.037607,42.035968,42.0365385,42.035287,42.035806,42.034747,42.035289,42.035331,42.034595,42.034581,42.037236,42.048416,42.048488,42.047338,42.047253,42.047101,42.046418,42.047171,42.043782,42.034509,42.047708,42.04896,42.04876,42.047652,42.046476,42.045806,42.045799,42.043236,42.043842,42.043673,42.044693,42.048796,42.0467129,42.046635,42.044821,42.043935,42.043135,42.044843,42.044774,42.04488,42.038492,42.0407264,42.040011,42.040888,42.041104,42.039463,42.038596,42.035489,42.03464,42.035951,42.036178,42.048294,42.046827,42.048608,42.04806,42.04579,42.044886,42.043966,42.043991,42.04297,42.042233,42.040931,42.040857,42.0404571,42.039674,42.0385265,42.038356,42.038398,42.037122,42.034918,42.034686,42.034754,42.036223,42.035829,42.034746,42.0380659,42.037679,42.037144,42.037328,42.036015,42.036002,42.034574,42.035257,42.0417879,42.04015,42.040295,42.04124,42.039047,42.040766,42.038811,42.037129,42.035712,42.035787,42.035841,42.037233,42.03602,42.035968,42.035706,42.037214,42.034858,42.037049,42.038948,42.038257,42.035287,42.0342224,42.0320748,42.0323259,42.032396,42.032202,42.032108,42.0319053,42.031305,42.0304742,42.0303538,42.030282,42.033381,42.033579,42.032048,42.032301,42.032227,42.030523,42.030196,42.0290352,42.028957,42.029305,42.028104,42.028556,42.0282044,42.027062,42.027227,42.023218,42.033558,42.0333765,42.031564,42.031542,42.033406,42.033616,42.03329,42.033347,42.03252,42.03246,42.031292,42.032375,42.03235,42.0310861,42.030062,42.028948,42.027854,42.027998,42.027039,42.026956,42.025306,42.023085,42.025796,42.024143,42.022828,42.032681,42.028101,42.028098,42.028049,42.0269586,42.022825,42.033633,42.032388,42.029863,42.030628,42.031138,42.033427,42.032302,42.032652,42.032564,42.032503,42.023229,42.028468,42.027656,42.027506,42.024483,42.023982,42.023916,42.026133,42.024993,42.0246851,42.023548,42.033714,42.0318449,42.03254,42.030625,42.030678,42.03252,42.031622,42.031834,42.031623,42.031545,42.0314669,42.0314606,42.0312822,42.0310946,42.0309301,42.026626,42.026482,42.026414,42.026372,42.025707,42.025293,42.024628,42.024613,42.024765,42.024327,42.024399,42.029772,42.0255464,42.021171,42.021176,42.021207,42.020817,42.018996,42.0197228,42.018597,42.017569,42.016903,42.018667,42.016862,42.016399,42.01571,42.016551,42.01613,42.015713,42.021384,42.020087,42.019697,42.019691,42.020666,42.020991,42.021788,42.020843,42.017109,42.018722,42.018847,42.01877,42.018703,42.016973,42.016978,42.0181479,42.018875,42.017856,42.016819,42.016797,42.016274,42.016271,42.016468,42.013957,42.013512,42.020123,42.021219,42.018988,42.0196717,42.017026,42.01896,42.019423,42.020321,42.0217997,42.021496,42.019004,42.019381,42.018592,42.022092,42.022091,42.020123,42.019713,42.017564,42.016921,42.016182,42.018931,42.017889,42.0189256,42.016835,42.016957,42.016067,42.0191252,42.019181,42.018707,42.017574,42.016684,42.0158534,42.013169,42.014943,42.013753,42.009489,42.009489,42.009489,42.009489,42.010636,42.01141,42.021425,42.021596,42.020184,42.008683,42.0183,41.996672,41.997119,41.99539,41.997067,41.997069,41.996919,41.996066,42.00138,41.995027,41.995058,41.997684,41.998952,41.9981656,41.995359,41.992964,41.9935397,41.992193,41.993128,41.992122,41.991786,41.992919,41.992185,41.9928219,41.99217,41.992539,41.991786,41.991365,41.987535,41.988776,41.988222,41.986509,41.991493,41.99095,42.0525525,42.051446,42.05996,42.061231,42.060771,42.060822,42.060369,42.060639,42.062976,42.061374,42.06076,42.059364,42.059411,42.058445,42.05701,42.059518,42.061394,42.060241,42.059792,42.0586718,42.056015,42.054514,42.054562,42.054439,42.055358,42.054503,42.054211,42.055432,42.052544,42.051473,42.050679,42.049449,42.04986,42.051074,42.056306,42.05539,42.054328,42.0564348,42.052794,42.052755,42.051479,42.052332,42.051678,42.051687,42.051659,42.050001,42.050257,42.04845,42.062046,42.0632692,42.063129,42.062259,42.0624143,42.061133,42.06027,42.060427,42.060425,42.0599011,42.058484,42.058683,42.058573,42.058649,42.057028,42.058694,42.057164,42.0579589,42.056766,42.057446,42.0571198,42.057468,42.057094,42.056616,42.062299,42.062285,42.062004,42.061544,42.061312,42.058377,42.057986,42.058595,42.059507,42.05695,42.058581,42.053879,42.053666,42.055748,42.055777,42.0522319,42.052272,42.050383,42.05162,42.049501,42.054236,42.055102,42.054366,42.054306,42.054319,42.055517,42.054227,42.053183,42.051646,42.05165,42.050307,42.052278,42.052456,42.0520487,42.05138,42.052173,42.051224,42.050899,42.037836,42.035271,42.035132,42.037918,42.037806,42.03805,42.036525,42.035875,42.035878,42.034988,42.036089,42.034535,42.036501,42.036466,42.036039,42.035179,42.034614,42.035703,42.03459,42.0375,42.034532,42.047666,42.048607,42.047467,42.048089,42.048191,42.047806,42.047571,42.047102,42.047103,42.046381,42.046377,42.046372,42.046085,42.046961,42.046962,42.046647,42.047171,42.047171,42.047171,42.045012,42.044642,42.036729,42.047928,42.049141,42.048762,42.046399,42.04378,42.0471033,42.046584,42.048185,42.044262,42.044058,42.044851,42.043845,42.044101,42.044773,42.044773,42.04066,42.040112,42.0417754,42.0417764,42.041811,42.038323,42.04102,42.0418699,42.041172,42.035571,42.034563,42.035741,42.0380577,42.036178,42.036102,42.03713,42.034679,42.048952,42.047782,42.0470858,42.0470799,42.046935,42.046572,42.044736,42.044791,42.044853,42.042187,42.042023,42.042304,42.040479,42.040301,42.040391,42.039531,42.0372039,42.03471,42.037302,42.035893,42.035723,42.035222,42.040021,42.040142,42.041173,42.038686,42.037104,42.035813,42.037162,42.037006,42.037291,42.034564,42.034789,42.034686,42.0347301,42.034636,42.036973,42.037025,42.0342174,42.034128,42.033379,42.032611,42.032419,42.030427,42.0304142,42.030418,42.030482,42.032161,42.032272,42.030406,42.0303597,42.033249,42.032008,42.032094,42.032381,42.0311,42.0304823,42.029028,42.02928,42.029281,42.028287,42.02793,42.027073,42.027032,42.0288284,42.028397,42.029209,42.0273991,42.0272517,42.02774,42.026691,42.03361,42.0343116,42.03338,42.03047,42.033456,42.033568,42.03337,42.033511,42.033321,42.033473,42.033366,42.033493,42.031354,42.031293,42.031493,42.031461,42.031496,42.031467,42.031432,42.031049,42.030388,42.031057,42.029399,42.028426,42.02903,42.029038,42.0255246,42.024111,42.0249938,42.0242588,42.022574,42.02295,42.026478,42.024885,42.0242331,42.034409,42.033423,42.02393,42.033604,42.033225,42.032365,42.031571,42.031248,42.031123,42.031348,42.03156,42.030552,42.034428,42.034428,42.033454,42.032495,42.032566,42.032155,42.032653,42.029453,42.024662,42.0243033,42.02521,42.028568,42.028595,42.027332,42.027507,42.028381,42.028188,42.0280297,42.025036,42.0236702,42.0244599,42.022902,42.022769,42.02283,42.031766,42.033912,42.03085,42.031077,42.030621,42.025411,42.027368,42.027193,42.0249247,42.025781,42.021173,42.01915,42.019153,42.018501,42.017423,42.017517,42.017914,42.017766,42.016276,42.01626,42.015949,42.022136,42.021234,42.019769,42.020694,42.02107,42.021275,42.020796,42.018412,42.018709,42.017598,42.018006,42.01727,42.016207,42.014435,42.016709,42.017866,42.016254,42.017854,42.016262,42.013952,42.013912,42.013641,42.013622,42.015927,42.015927,42.016121,42.016122,42.01975,42.019937,42.021244,42.019997,42.019996,42.02099,42.019861,42.018991,42.018852,42.01883,42.018898,42.0202945,42.020394,42.019544,42.020112,42.021582,42.01797,42.017962,42.019959,42.018541,42.017874,42.02224,42.022238,42.0214496,42.019986,42.017413,42.017083,42.019272,42.019092,42.019005,42.01791,42.017834,42.0176853,42.01789,42.0168678,42.016072,42.019833,42.018443,42.018888,42.018766,42.016769,42.015667,42.014567,42.012638,42.012488,42.009489,42.009489,42.022607,42.021806,42.022516,42.020379,42.021389,42.018806,42.021162,42.021594,42.022626,42.009326,42.008803,42.008806,42.0091899,41.99507,41.997124,41.997062,41.997069,41.996903,42.00514,41.998999,41.994841,41.99373,41.995142,41.993751,41.995175,41.999812,41.999575,41.9976055,41.995512,41.99639,41.99423,41.993611,41.99338,41.990054,41.993059,41.993185,41.99261,41.991372,41.991883,41.991779,41.992133,41.991933,41.988675,41.987109,41.988934,41.98811,41.987289,41.98658,41.991885,42.022745,42.051297,42.051357,42.049671,42.050443,42.060466,42.061664,42.061326,42.063067,42.059592,42.059517,42.059242,42.059293,42.05817,42.057533,42.057022,42.059375,42.059309,42.060431,42.060144,42.060096,42.059743,42.058998,42.058996,42.054368,42.053916,42.055545,42.0545,42.054941,42.055217,42.054427,42.05622,42.052353,42.049832,42.050075,42.050372,42.049956,42.051039,42.048765,42.055264,42.053407,42.054612,42.053547,42.0562952,42.056395,42.05467,42.054789,42.055319,42.052669,42.052698,42.052696,42.052294,42.051833,42.05057,42.0488205,42.062574,42.063296,42.0632722,42.063141,42.0619544,42.061976,42.0619709,42.061284,42.062259,42.0628153,42.0624082,42.062434,42.061504,42.06027,42.060269,42.060576,42.060427,42.059561,42.06113,42.059677,42.058634,42.058619,42.059211,42.059202,42.059197,42.059194,42.059175,42.059176,42.059176,42.05826,42.058276,42.058284,42.056827,42.057124,42.05738,42.0568529,42.057209,42.063388,42.062483,42.062279,42.062887,42.062931,42.062974,42.063019,42.063106,42.063342,42.062584,42.062581,42.062946,42.063301,42.061713,42.061429,42.061483,42.061541,42.061531,42.061392,42.060021,42.059181,42.057981,42.058401,42.059526,42.0533209,42.055751,42.05359,42.053467,42.05198,42.052361,42.052247,42.049509,42.054408,42.055757,42.054318,42.054177,42.054417,42.054354,42.054154,42.054153,42.054153,42.054157,42.053185,42.053108,42.053032,42.053067,42.051802,42.051655,42.050206,42.050301,42.052269,42.051377,42.052267,42.052146,42.051054,42.037586,42.037643,42.035259,42.036038,42.036464,42.034569,42.035678,42.0362103,42.034678,42.035871,42.035857,42.034532,42.034724,42.035675,42.034652,42.034819,42.037502,42.034644,42.047384,42.045776,42.047618,42.0474249,42.047406,42.047705,42.047699,42.047687,42.047667,42.047104,42.047104,42.046144,42.047171,42.046137,42.046135,42.046411,42.044383,42.044989,42.042994,42.043754,42.043048,42.042961,42.037103,42.0349702,42.046549,42.048067,42.04849,42.047615,42.043363,42.043005,42.04259,42.04389,42.046504,42.043121,42.044036,42.042485,42.042479,42.044921,42.043609,42.039141,42.040143,42.0417369,42.0413937,42.04074,42.041245,42.041332,42.039212,42.0392,42.035684,42.036023,42.0362,42.03465,42.034906,42.049216,42.048934,42.045809,42.0459325,42.046882,42.044053,42.044735,42.043932,42.043946,42.044218,42.043952,42.043233,42.0426969,42.0418697,42.041243,42.040451,42.03981,42.039369,42.03846,42.0384781,42.034789,42.0347206,42.034812,42.036077,42.036027,42.03725,42.0380982,42.0357,42.035779,42.035854,42.034878,42.039338,42.039983,42.040146,42.038888,42.035733,42.035921,42.035876,42.037106,42.035862,42.036002,42.037084,42.034685,42.034614,42.036949,42.035423,42.0346,42.035034,42.0314733,42.03131,42.031369,42.0311108,42.033405,42.032343,42.033385,42.033489,42.033313,42.032276,42.030579,42.030311,42.03013,42.0299846,42.029218,42.029198,42.029223,42.0280396,42.028076,42.02819,42.027154,42.0282,42.0277574,42.026089,42.0268089,42.0272605,42.0272703,42.033311,42.034339,42.031846,42.030823,42.033482,42.033612,42.033336,42.032399,42.031433,42.031438,42.032386,42.031321,42.031483,42.0314873,42.0304477,42.029451,42.02752,42.0275775,42.025229,42.021576,42.0249982,42.0227,42.0238856,42.022963,42.0238391,42.026723,42.0261614,42.022951,42.033524,42.033285,42.028048,42.028004,42.033272,42.032426,42.034429,42.033193,42.032385,42.032098,42.031533,42.031215,42.029679,42.031139,42.029786,42.034422,42.032528,42.034421,42.033557,42.032638,42.033726,42.034018,42.02868,42.026185,42.023229,42.023021,42.028424,42.028236,42.024949,42.025818,42.0205201,42.020186,42.0246868,42.023874,42.023945,42.023451,42.022824,42.034453,42.033662,42.033576,42.032467,42.031566,42.030801,42.02667,42.026642,42.025666,42.025707,42.025594,42.025264,42.024598,42.024594,42.024633,42.025726,42.025156,42.024635,42.024485,42.029803,42.027956,42.027969,42.024305,42.022228,42.021324,42.020966,42.020912,42.019083,42.019028,42.018378,42.017907,42.016896,42.016829,42.016297,42.016214,42.021992,42.021386,42.022239,42.01972,42.021163,42.021056,42.021025,42.020861,42.021737,42.018518,42.018424,42.018849,42.016973,42.016961,42.016212,42.0172,42.0185298,42.018017,42.017856,42.017753,42.015974,42.014823,42.013576,42.013544,42.01343,42.013467,42.01469,42.0210331,42.020703,42.021221,42.021195,42.02103,42.020893,42.019926,42.0196959,42.019902,42.01917,42.018873,42.0164532,42.016642,42.018965,42.020062,42.020783,42.020504,42.022655,42.021436,42.017962,42.018943,42.01873,42.019937,42.018039,42.017106,42.0162813,42.016216,42.016217,42.016217,42.016853,42.019952,42.018796,42.020037,42.018504,42.018941,42.017573,42.016961,42.01629,42.0145503,42.012488,42.009321,42.011343,42.010333,42.010076,42.022836,42.022474,42.022658,42.022566,42.022685,42.0228,42.021547,42.021278,42.021497,42.021555,42.009401,42.009401,41.994357,41.994207,41.994206,41.995218,41.995973,41.995512,42.001694,41.994241,41.994137,41.9941,41.994824,41.995088,41.997992,41.997364,41.997126,41.996089,41.995508,41.997299,41.995341,41.993986,41.997352,41.993288,41.992975,41.993356,41.9927922,41.991211,41.992917,41.99284,41.991709,41.991641,41.991868,41.991612,41.991098,41.989581,41.990264,41.989905,41.987606,41.989768,41.98822,41.986588,42.053327,42.052519,42.050507,42.059909,42.06048,42.060736,42.062828,42.06281,42.062802,42.062956,42.060625,42.0591,42.058421,42.058201,42.059599,42.059332,42.058136,42.060527,42.060516,42.061478,42.061008,42.059957,42.061096,42.0593867,42.056702,42.058563,42.058816,42.055607,42.054367,42.0543376,42.054956,42.0562619,42.05327,42.0524029,42.053418,42.049771,42.052522,42.052359,42.04977,42.049608,42.055131,42.053459,42.054385,42.055319,42.051798,42.051839,42.051673,42.051664,42.048947,42.050705,42.061472,42.063307,42.061901,42.0628074,42.062109,42.061134,42.060342,42.060866,42.060424,42.060357,42.060377,42.061544,42.061377,42.058544,42.05869,42.059216,42.059017,42.059184,42.057164,42.058238,42.057164,42.057164,42.05758,42.062361,42.061915,42.061987,42.063329,42.063185,42.062374,42.0623,42.063301,42.063301,42.063301,42.062831,42.062873,42.060239,42.060293,42.059803,42.060074,42.05865,42.05853,42.057905,42.057838,42.05696,42.058461,42.053314,42.055601,42.053466,42.05178,42.049423,42.051073,42.049687,42.049528,42.055056,42.054176,42.054175,42.053183,42.053033,42.0518,42.052265,42.052272,42.052346,42.051355,42.051426,42.051382,42.051245,42.051225,42.051224,42.037843,42.036149,42.037915,42.037911,42.037827,42.036605,42.036849,42.03674,42.035629,42.03748,42.035235,42.036087,42.036206,42.035302,42.035094,42.035696,42.035841,42.035325,42.03457,42.03458,42.034663,42.037065,42.045763,42.048608,42.047997,42.048058,42.047592,42.04768,42.047665,42.047642,42.047618,42.047595,42.046145,42.046429,42.044609,42.042369,42.042826,42.043351,42.037409,42.045632,42.048199,42.048997,42.046472,42.04806,42.044669,42.046496,42.045706,42.045728,42.045728,42.045885,42.045878,42.04795,42.04657,42.045685,42.044308,42.043214,42.044978,42.042114,42.040072,42.038434,42.04102,42.0414084,42.041314,42.040273,42.040143,42.039202,42.035836,42.035379,42.0380632,42.034612,42.049031,42.048528,42.0422973,42.042156,42.043129,42.044735,42.04498,42.044975,42.044254,42.0427,42.042905,42.041821,42.040252,42.039387,42.038602,42.0383475,42.038836,42.040276,42.039351,42.034934,42.034617,42.0378602,42.037001,42.037097,42.037041,42.0367836,42.035857,42.034574,42.035277,42.034761,42.040944,42.040811,42.040105,42.040081,42.040297,42.040148,42.0385,42.041874,42.03902,42.036098,42.035996,42.036099,42.036053,42.034805,42.03471,42.034662,42.034829,42.034757,42.034708,42.036792,42.03548,42.034753,42.0345614,42.0334994,42.032346,42.03139,42.031506,42.031347,42.030396,42.031196,42.033623,42.032305,42.030267,42.032064,42.032267,42.031964,42.030368,42.03146,42.031138,42.029136,42.029281,42.0281148,42.027953,42.028963,42.029025,42.027992,42.0272703,42.03179,42.033306,42.033369,42.032299,42.031499,42.032396,42.031415,42.031436,42.032417,42.0310818,42.030389,42.028544,42.026886,42.02689,42.025258,42.025459,42.0249549,42.0242386,42.023117,42.0238089,42.034576,42.033296,42.0331453,42.0277962,42.028002,42.027997,42.033521,42.0333,42.032383,42.031356,42.032216,42.030702,42.031362,42.031347,42.030231,42.030232,42.033285,42.032291,42.032307,42.033686,42.032478,42.031635,42.032157,42.032575,42.029453,42.028836,42.023651,42.024705,42.02653,42.027664,42.026614,42.02821,42.028191,42.026458,42.025102,42.025197,42.026308,42.0247523,42.022824,42.032082,42.032493,42.030933,42.024601,42.024555,42.024615,42.025218,42.025194,42.02437,42.023306,42.023308,42.029884,42.028168,42.025573,42.02438,42.024592,42.022227,42.021209,42.0209,42.018123,42.017631,42.01686,42.016217,42.016913,42.020145,42.02007,42.021811,42.021889,42.021785,42.01834,42.018664,42.017055,42.018188,42.016979,42.016904,42.01869,42.017856,42.017867,42.017858,42.016252,42.016618,42.015914,42.016092,42.014636,42.014109,42.015793,42.016004,42.01397,42.015765,42.013519,42.014593,42.0195855,42.020995,42.020342,42.020592,42.020547,42.01899,42.021168,42.018648,42.018961,42.01896,42.018959,42.018813,42.018812,42.01881,42.018809,42.02011,42.022676,42.022659,42.021432,42.018848,42.017727,42.022089,42.021559,42.020871,42.020286,42.018837,42.018282,42.01924,42.01895,42.018808,42.018857,42.016941,42.0165794,42.016072,42.017609,42.01874,42.019759,42.017131,42.016827,42.016385,42.017589,42.0157866,42.016124,42.013296,42.013495,42.01359,42.011793,42.022786,42.022709,42.020441,42.020855,42.020207,42.02153,42.019025,42.019349,42.019099,42.022603,42.009398,42.008375,41.996849,41.994357,41.995987,41.996907,41.995702,41.996063,42.004509,41.997906,41.999553,41.998023,41.999342,41.994347,41.994229,41.99222,41.99329,41.993173,41.99315,41.990054,41.990993,41.990523,41.993062,41.991574,41.992303,41.992159,41.991708,41.992522,41.99171,41.990134,41.9887374,41.987686,41.98651,41.990921],[-93.6193873,-93.61732,-93.638933,-93.638925,-93.633792,-93.633826,-93.632852,-93.639068,-93.636947,-93.638647,-93.634626,-93.632913,-93.62655,-93.6235954,-93.622971,-93.636655,-93.636372,-93.639366,-93.626537,-93.628806,-93.627112,-93.622769,-93.624373,-93.627271,-93.62728,-93.627232,-93.625924,-93.625966,-93.625848,-93.653201,-93.654645,-93.6537329,-93.652713,-93.652336,-93.655051,-93.656067,-93.654853,-93.654121,-93.649447,-93.651013,-93.644356,-93.641635,-93.642252,-93.651381,-93.652334,-93.654206,-93.653616,-93.650578,-93.652972,-93.642368,-93.650318,-93.647105,-93.642224,-93.692412,-93.68981,-93.687694,-93.686263,-93.686264,-93.68628,-93.685028,-93.685029,-93.677205,-93.673669,-93.672096,-93.672067,-93.660672,-93.651773,-93.652773,-93.647988,-93.64614,-93.645599,-93.644889,-93.644889,-93.64489,-93.6468879,-93.641253,-93.643556,-93.648143,-93.65921,-93.639516,-93.633351,-93.637463,-93.637788,-93.631435,-93.632554,-93.626137,-93.625926,-93.623383,-93.626939,-93.627536,-93.625728,-93.622904,-93.622864,-93.620651,-93.620504,-93.629287,-93.629313,-93.62475,-93.624601,-93.621431,-93.614244,-93.617553,-93.617117,-93.6177041,-93.616582,-93.614882,-93.614241,-93.613624,-93.615622,-93.617092,-93.61722,-93.618428,-93.615475,-93.611642,-93.6192014,-93.620341,-93.6116762,-93.607275,-93.606905,-93.61061,-93.605969,-93.609218,-93.607801,-93.605943,-93.606742,-93.604835,-93.603705,-93.617014,-93.62024,-93.620241,-93.6179112,-93.615431,-93.608954,-93.6090761,-93.610466,-93.610468,-93.606789,-93.606789,-93.610588,-93.61221,-93.607238,-93.608752,-93.607088,-93.625737,-93.625586,-93.622597,-93.623523,-93.622448,-93.620409,-93.62338,-93.621674,-93.621835,-93.621632,-93.629495,-93.626876,-93.620386,-93.656231,-93.658265,-93.656924,-93.658081,-93.657573,-93.672036,-93.671021,-93.671076,-93.673401,-93.672381,-93.678141,-93.676659,-93.677992,-93.676654,-93.676674,-93.673565,-93.672523,-93.670503,-93.670114,-93.663619,-93.667904,-93.675038,-93.671178,-93.662762,-93.66357,-93.667708,-93.66526,-93.6658876,-93.663367,-93.659808,-93.685988,-93.683167,-93.682026,-93.69019,-93.69245,-93.692521,-93.68329,-93.6801805,-93.682138,-93.69188,-93.688258,-93.68773,-93.687807,-93.687745,-93.683727,-93.687859,-93.685523,-93.687764,-93.680864,-93.682686,-93.681227,-93.6837,-93.687922,-93.685961,-93.6783313,-93.677482,-93.677545,-93.676372,-93.675268,-93.675175,-93.6663375,-93.66751,-93.664841,-93.655762,-93.655653,-93.6514413,-93.64678,-93.646683,-93.639781,-93.642793,-93.639678,-93.645014,-93.645729,-93.641677,-93.625291,-93.629496,-93.626689,-93.615272,-93.608271,-93.60775,-93.615993,-93.604947,-93.602823,-93.603972,-93.602002,-93.657894,-93.649203,-93.649096,-93.649471,-93.647645,-93.646814,-93.652495,-93.60743,-93.608372,-93.60359,-93.604318,-93.604425,-93.603435,-93.601806,-93.603398,-93.601449,-93.606856,-93.6035354,-93.600582,-93.599791,-93.600006,-93.600147,-93.619562,-93.639388,-93.6364,-93.637442,-93.633024,-93.636955,-93.637046,-93.637671,-93.635824,-93.635753,-93.636822,-93.636047,-93.634644,-93.633245,-93.628639,-93.629357,-93.636512,-93.637774,-93.63745,-93.638366,-93.634212,-93.637043,-93.637289,-93.635919,-93.639511,-93.6392308,-93.639366,-93.632915,-93.630253,-93.631432,-93.631791,-93.625856,-93.626554,-93.626603,-93.627037,-93.625657,-93.625542,-93.6234986,-93.6223289,-93.627396,-93.628119,-93.629456,-93.629851,-93.629881,-93.624729,-93.625945,-93.625918,-93.625846,-93.625688,-93.62569,-93.625696,-93.6256036,-93.626521,-93.658877,-93.6562749,-93.652716,-93.657828,-93.654241,-93.654144,-93.653955,-93.652863,-93.654439,-93.658224,-93.657835,-93.657049,-93.654437,-93.658854,-93.656737,-93.653152,-93.652884,-93.654724,-93.652084,-93.652724,-93.6541267,-93.6547754,-93.643771,-93.65604,-93.6545189,-93.65048,-93.649779,-93.649743,-93.650244,-93.655483,-93.654215,-93.654405,-93.6514849,-93.650238,-93.645752,-93.641561,-93.642094,-93.642911,-93.642914,-93.642749,-93.643312,-93.644106,-93.642626,-93.640051,-93.640065,-93.649649,-93.642813,-93.642932,-93.641919,-93.641377,-93.641313,-93.641303,-93.641397,-93.641312,-93.640879,-93.65429,-93.657163,-93.650957,-93.652608,-93.65238,-93.655555,-93.655988,-93.654048,-93.651503,-93.652117,-93.650529,-93.650822,-93.644121,-93.64247,-93.642553,-93.643944,-93.641251,-93.641296,-93.6504,-93.650417,-93.650377,-93.650338,-93.648275,-93.64287,-93.643842,-93.642474,-93.643752,-93.641501,-93.692017,-93.692472,-93.691506,-93.691707,-93.691068,-93.690159,-93.690402,-93.689136,-93.6888,-93.687843,-93.686267,-93.687804,-93.685037,-93.6790752,-93.680198,-93.676364,-93.677008,-93.675716,-93.669538,-93.673518,-93.669845,-93.660327,-93.646316,-93.646338,-93.647039,-93.646916,-93.644205,-93.645482,-93.641799,-93.645923,-93.65921,-93.635342,-93.634803,-93.631881,-93.631519,-93.633163,-93.632537,-93.632823,-93.637964,-93.637905,-93.632647,-93.631899,-93.628903,-93.627047,-93.624612,-93.630714,-93.627416,-93.622991,-93.623485,-93.622635,-93.623411,-93.628658,-93.6273127,-93.627836,-93.626979,-93.624581,-93.622685,-93.621699,-93.629499,-93.625633,-93.622644,-93.622494,-93.617963,-93.615686,-93.615602,-93.614619,-93.614216,-93.619029,-93.618784,-93.61488,-93.614768,-93.61588,-93.620016,-93.618478,-93.6165473,-93.618549,-93.6190376,-93.617219,-93.6124,-93.617207,-93.618605,-93.618607,-93.62008,-93.620336,-93.61719,-93.617182,-93.6146206,-93.6144209,-93.61239,-93.612241,-93.614049,-93.615465,-93.613899,-93.612253,-93.610355,-93.607655,-93.610588,-93.610649,-93.609226,-93.606573,-93.60552,-93.609059,-93.608977,-93.61061,-93.608903,-93.60891,-93.606891,-93.606889,-93.605968,-93.606786,-93.610605,-93.604987,-93.604047,-93.603879,-93.603701,-93.6179738,-93.619013,-93.6195025,-93.62026,-93.615598,-93.615599,-93.6195361,-93.618486,-93.6200683,-93.6176367,-93.615381,-93.608866,-93.607776,-93.608829,-93.606842,-93.606841,-93.610469,-93.606788,-93.6178679,-93.613887,-93.612245,-93.610582,-93.610044,-93.6076506,-93.608704,-93.607058,-93.605207,-93.628848,-93.6279815,-93.625968,-93.626116,-93.624564,-93.624567,-93.622513,-93.623515,-93.623645,-93.623644,-93.623629,-93.622565,-93.621351,-93.6213592,-93.62726,-93.621525,-93.621504,-93.621506,-93.621635,-93.621633,-93.628299,-93.626716,-93.621862,-93.625154,-93.599575,-93.659205,-93.658207,-93.658173,-93.657107,-93.6601886,-93.657241,-93.675401,-93.673382,-93.677991,-93.671849,-93.671369,-93.666432,-93.666813,-93.662126,-93.659279,-93.660877,-93.671177,-93.666242,-93.665996,-93.665994,-93.667776,-93.668212,-93.667209,-93.665271,-93.663366,-93.6641939,-93.661005,-93.685733,-93.685986,-93.68341,-93.681063,-93.682176,-93.679789,-93.679784,-93.6791403,-93.679109,-93.679784,-93.6795273,-93.6790969,-93.6791456,-93.6791251,-93.679181,-93.687067,-93.688859,-93.688993,-93.689091,-93.690266,-93.690971,-93.689072,-93.691433,-93.69157,-93.690111,-93.689066,-93.685125,-93.6825646,-93.691883,-93.692135,-93.688372,-93.690401,-93.691585,-93.6928668,-93.690548,-93.690324,-93.691667,-93.688383,-93.689431,-93.691175,-93.689438,-93.689942,-93.691672,-93.688628,-93.686916,-93.687212,-93.685068,-93.684954,-93.683157,-93.683159,-93.68317,-93.68351,-93.687819,-93.686218,-93.68269,-93.682623,-93.682983,-93.686744,-93.685673,-93.67915,-93.680706,-93.680782,-93.68218,-93.68167,-93.681297,-93.681439,-93.684115,-93.685333,-93.684536,-93.678305,-93.676372,-93.672234,-93.6739723,-93.676608,-93.669952,-93.666315,-93.664685,-93.6644556,-93.659191,-93.664812,-93.664336,-93.663302,-93.657963,-93.655674,-93.651825,-93.652027,-93.651786,-93.64857,-93.651669,-93.648414,-93.646655,-93.6459433,-93.648419,-93.644704,-93.644636,-93.6428055,-93.6397,-93.642371,-93.640592,-93.644295,-93.6439975,-93.642885,-93.640762,-93.641356,-93.645743,-93.645743,-93.645743,-93.645743,-93.644864,-93.643539,-93.626839,-93.625299,-93.616895,-93.6165709,-93.588227,-93.607252,-93.607055,-93.603717,-93.604716,-93.604491,-93.604519,-93.602424,-93.652119,-93.65315,-93.651832,-93.646455,-93.644884,-93.6446004,-93.646816,-93.651703,-93.6042638,-93.608359,-93.608196,-93.60821,-93.604252,-93.604218,-93.603479,-93.600636,-93.603111,-93.601975,-93.60176,-93.601127,-93.606688,-93.604928,-93.603867,-93.606925,-93.599866,-93.600191,-93.6188286,-93.6163289,-93.639062,-93.637796,-93.638775,-93.638778,-93.637482,-93.636718,-93.633876,-93.633781,-93.633797,-93.639007,-93.637763,-93.636798,-93.637279,-93.632013,-93.628831,-93.627911,-93.622299,-93.6225667,-93.6391494,-93.639715,-93.637825,-93.637808,-93.636511,-93.636505,-93.630299,-93.633168,-93.635153,-93.63719,-93.633905,-93.635216,-93.6341769,-93.630231,-93.626544,-93.627875,-93.630208,-93.624265,-93.628971,-93.627629,-93.627114,-93.629462,-93.62765,-93.627309,-93.629425,-93.627234,-93.625848,-93.62649,-93.658873,-93.6573126,-93.652941,-93.654642,-93.6566551,-93.657889,-93.657188,-93.656801,-93.654734,-93.654443,-93.656958,-93.65572,-93.652549,-93.651216,-93.655587,-93.654914,-93.654187,-93.6513669,-93.653868,-93.651845,-93.6514243,-93.649997,-93.650607,-93.65089,-93.640958,-93.6424399,-93.643039,-93.643964,-93.640051,-93.649587,-93.641331,-93.642397,-93.643632,-93.641342,-93.640959,-93.654594,-93.65412,-93.653324,-93.650932,-93.651562,-93.655279,-93.656036,-93.653929,-93.65579,-93.644107,-93.642334,-93.642092,-93.642376,-93.643976,-93.64403,-93.644024,-93.643788,-93.650397,-93.650358,-93.648255,-93.643836,-93.64119,-93.6430423,-93.643692,-93.641651,-93.644018,-93.639662,-93.692497,-93.692431,-93.691695,-93.691836,-93.690761,-93.690364,-93.691246,-93.689769,-93.689024,-93.689801,-93.688803,-93.687819,-93.686428,-93.686278,-93.674093,-93.675743,-93.673965,-93.672503,-93.670551,-93.66203,-93.660522,-93.653482,-93.653232,-93.647488,-93.646063,-93.646113,-93.64489,-93.644891,-93.644172,-93.644118,-93.646984,-93.646986,-93.646989,-93.647044,-93.643746,-93.643729,-93.643378,-93.641785,-93.64177,-93.641742,-93.647655,-93.648954,-93.655582,-93.635095,-93.634976,-93.631526,-93.631779,-93.631245,-93.6261431,-93.624762,-93.624632,-93.629079,-93.630152,-93.630732,-93.624499,-93.625799,-93.622673,-93.622662,-93.629555,-93.629884,-93.6277098,-93.6267969,-93.626171,-93.626762,-93.623267,-93.6219226,-93.621566,-93.62936,-93.627578,-93.628236,-93.6285065,-93.622644,-93.622644,-93.623556,-93.620487,-93.6167,-93.616552,-93.6178602,-93.6169916,-93.615685,-93.615614,-93.619022,-93.617544,-93.617543,-93.614856,-93.612394,-93.613383,-93.617238,-93.617084,-93.617086,-93.615621,-93.6165173,-93.618457,-93.615484,-93.61076,-93.612269,-93.612253,-93.607422,-93.610663,-93.605189,-93.605659,-93.608909,-93.60688,-93.605971,-93.60597,-93.606788,-93.607948,-93.608765,-93.608903,-93.6071073,-93.607799,-93.604986,-93.604837,-93.6195726,-93.619415,-93.615582,-93.62005,-93.616991,-93.6189373,-93.6195822,-93.6198478,-93.620219,-93.613969,-93.615447,-93.615031,-93.6112887,-93.607771,-93.606764,-93.607713,-93.607724,-93.606802,-93.6077128,-93.61844,-93.620183,-93.618297,-93.617386,-93.616885,-93.616998,-93.616996,-93.6096888,-93.610008,-93.608752,-93.6075521,-93.6059697,-93.606196,-93.604486,-93.628806,-93.6267116,-93.625728,-93.625926,-93.624715,-93.625584,-93.623666,-93.624566,-93.623514,-93.621532,-93.621532,-93.622102,-93.624688,-93.624687,-93.623632,-93.622552,-93.621485,-93.620392,-93.620391,-93.624591,-93.624471,-93.6218858,-93.627,-93.625556,-93.621676,-93.621065,-93.6259506,-93.628447,-93.6261621,-93.6265884,-93.6295,-93.625289,-93.623738,-93.624819,-93.6246096,-93.656979,-93.657958,-93.658395,-93.675551,-93.669806,-93.672241,-93.676873,-93.677717,-93.677945,-93.672718,-93.674179,-93.671905,-93.666954,-93.666912,-93.665521,-93.665355,-93.665356,-93.666979,-93.660812,-93.678125,-93.676219,-93.6739889,-93.67358,-93.666066,-93.666216,-93.666147,-93.66515,-93.663054,-93.660595,-93.6608625,-93.66526,-93.6644616,-93.6618779,-93.662924,-93.658528,-93.659949,-93.683223,-93.683302,-93.682029,-93.682032,-93.682175,-93.691284,-93.68698,-93.686789,-93.6826513,-93.679198,-93.692009,-93.691213,-93.691612,-93.690719,-93.690276,-93.690113,-93.688995,-93.689019,-93.691706,-93.688958,-93.68895,-93.685742,-93.686888,-93.685158,-93.685865,-93.6844,-93.684289,-93.683158,-93.687704,-93.684227,-93.684396,-93.682718,-93.685572,-93.686982,-93.687674,-93.683856,-93.67994,-93.68369,-93.679505,-93.68031,-93.686931,-93.685207,-93.684451,-93.684397,-93.683127,-93.682991,-93.681436,-93.681401,-93.678303,-93.678304,-93.677407,-93.673447,-93.673571,-93.673137,-93.671443,-93.672408,-93.675294,-93.673251,-93.670627,-93.6649923,-93.6653819,-93.665562,-93.661895,-93.660115,-93.664925,-93.665829,-93.663479,-93.66345,-93.661763,-93.6558,-93.656681,-93.655851,-93.64856,-93.651785,-93.651635,-93.647024,-93.646748,-93.648411,-93.646438,-93.646438,-93.6447083,-93.646288,-93.6447134,-93.646735,-93.642886,-93.641405,-93.63962,-93.639497,-93.641084,-93.643378,-93.64293,-93.640311,-93.640293,-93.645743,-93.645743,-93.628409,-93.627459,-93.625288,-93.626835,-93.625143,-93.625217,-93.615426,-93.616849,-93.604193,-93.616242,-93.616584,-93.615985,-93.615921,-93.607342,-93.607121,-93.60489,-93.604549,-93.602912,-93.639793,-93.650229,-93.653025,-93.651718,-93.649198,-93.649475,-93.650431,-93.644679,-93.646054,-93.6470597,-93.646726,-93.646645,-93.647834,-93.647214,-93.608267,-93.610145,-93.608197,-93.606374,-93.605351,-93.604649,-93.603601,-93.604269,-93.603178,-93.603458,-93.610109,-93.6051072,-93.604761,-93.603917,-93.603886,-93.608291,-93.600066,-93.577427,-93.618504,-93.618503,-93.617273,-93.61728,-93.63736,-93.636146,-93.636972,-93.634341,-93.636345,-93.637883,-93.639068,-93.637848,-93.637917,-93.635833,-93.633329,-93.634513,-93.631622,-93.629016,-93.629075,-93.622874,-93.629538,-93.628754,-93.628723,-93.637875,-93.638853,-93.637876,-93.637816,-93.634665,-93.633973,-93.631528,-93.632669,-93.638554,-93.634714,-93.634581,-93.636497,-93.633898,-93.633757,-93.632191,-93.626331,-93.629118,-93.626722,-93.627112,-93.6247461,-93.621646,-93.624798,-93.624798,-93.624431,-93.629747,-93.628656,-93.627855,-93.629777,-93.627242,-93.624575,-93.6257971,-93.6588809,-93.658015,-93.6566403,-93.654243,-93.6570383,-93.6566568,-93.6562702,-93.657967,-93.654774,-93.6570542,-93.6562861,-93.657444,-93.65331,-93.657211,-93.657235,-93.654689,-93.654617,-93.650917,-93.657492,-93.6565389,-93.65699,-93.6545189,-93.650597,-93.650499,-93.65045,-93.650406,-93.649838,-93.649808,-93.649798,-93.649397,-93.649319,-93.649291,-93.653414,-93.652565,-93.652054,-93.6514105,-93.650608,-93.646937,-93.645875,-93.64573,-93.644377,-93.644376,-93.644375,-93.644374,-93.644447,-93.644366,-93.641795,-93.641716,-93.640513,-93.642177,-93.644256,-93.644064,-93.643062,-93.643961,-93.643954,-93.639977,-93.64021,-93.64334,-93.641279,-93.642836,-93.643641,-93.6575919,-93.652894,-93.652641,-93.649901,-93.657271,-93.655159,-93.655905,-93.652946,-93.644118,-93.643769,-93.642069,-93.64176,-93.642261,-93.642243,-93.644086,-93.643836,-93.64248,-93.642344,-93.643544,-93.641251,-93.643748,-93.642275,-93.650341,-93.650299,-93.647584,-93.647933,-93.643974,-93.643792,-93.642046,-93.641233,-93.639574,-93.69179,-93.68912,-93.689767,-93.688804,-93.686428,-93.686413,-93.685185,-93.6791142,-93.677007,-93.673987,-93.673889,-93.669559,-93.669552,-93.672318,-93.673518,-93.671941,-93.661848,-93.661983,-93.653082,-93.652658,-93.647974,-93.64733,-93.647166,-93.646637,-93.646607,-93.646576,-93.646478,-93.644107,-93.644096,-93.645671,-93.641813,-93.639931,-93.639878,-93.643456,-93.648204,-93.645589,-93.64577,-93.648172,-93.645547,-93.645544,-93.658232,-93.6574973,-93.634545,-93.634528,-93.635899,-93.631554,-93.638051,-93.63687,-93.631673,-93.631302,-93.625898,-93.624745,-93.623104,-93.621344,-93.62232,-93.622834,-93.621452,-93.626945,-93.630023,-93.6279875,-93.6276375,-93.624484,-93.62326,-93.621368,-93.622804,-93.621503,-93.629545,-93.628654,-93.623704,-93.621485,-93.620497,-93.619383,-93.61706,-93.616099,-93.618143,-93.615535,-93.618786,-93.619352,-93.617521,-93.615281,-93.613046,-93.612347,-93.613457,-93.613983,-93.6192677,-93.617242,-93.615622,-93.618428,-93.617075,-93.612401,-93.612075,-93.618606,-93.6197518,-93.620343,-93.618445,-93.615616,-93.615485,-93.6133009,-93.615536,-93.612418,-93.6139,-93.613899,-93.607117,-93.609148,-93.610561,-93.605388,-93.610609,-93.610611,-93.607963,-93.606932,-93.605968,-93.60674,-93.606782,-93.605942,-93.606746,-93.604836,-93.603701,-93.603703,-93.604562,-93.6190545,-93.62024,-93.617135,-93.6176729,-93.615432,-93.615446,-93.60861,-93.608868,-93.606857,-93.606842,-93.61047,-93.606789,-93.618662,-93.6190848,-93.617061,-93.618294,-93.616911,-93.6188503,-93.617037,-93.612061,-93.612191,-93.608882,-93.6094532,-93.608678,-93.6047622,-93.6041765,-93.6051157,-93.6281873,-93.6262716,-93.627605,-93.627487,-93.624715,-93.623521,-93.621501,-93.623643,-93.623631,-93.624539,-93.621502,-93.6224,-93.622402,-93.6212119,-93.6209132,-93.62576,-93.622147,-93.6220032,-93.628447,-93.629496,-93.6277573,-93.629501,-93.6274918,-93.628263,-93.6263765,-93.622793,-93.62188,-93.623814,-93.655699,-93.656124,-93.657089,-93.655924,-93.675549,-93.675559,-93.669741,-93.674417,-93.67369,-93.677504,-93.676953,-93.675714,-93.677997,-93.677601,-93.676681,-93.668544,-93.664037,-93.662239,-93.660671,-93.660691,-93.661983,-93.661985,-93.676272,-93.669606,-93.671221,-93.671511,-93.66504,-93.663415,-93.66712,-93.66061,-93.6644275,-93.66468,-93.6643138,-93.658545,-93.66057,-93.659659,-93.658537,-93.693153,-93.684102,-93.684102,-93.683309,-93.683226,-93.681061,-93.686828,-93.686941,-93.690342,-93.69067,-93.691248,-93.691213,-93.689071,-93.689071,-93.69003,-93.688941,-93.688923,-93.689884,-93.688917,-93.684944,-93.682439,-93.679216,-93.683612,-93.691971,-93.692069,-93.690402,-93.690545,-93.692696,-93.692415,-93.690633,-93.688952,-93.689344,-93.689518,-93.691469,-93.692052,-93.684962,-93.686678,-93.687734,-93.684837,-93.684993,-93.683235,-93.684354,-93.683158,-93.683169,-93.687856,-93.686575,-93.682805,-93.686795,-93.687786,-93.68675,-93.683813,-93.6800822,-93.679805,-93.681274,-93.679505,-93.687684,-93.684137,-93.684714,-93.684625,-93.684285,-93.684271,-93.683732,-93.6783521,-93.677545,-93.676222,-93.671455,-93.671454,-93.671453,-93.673197,-93.6747048,-93.67525,-93.674327,-93.673253,-93.6762195,-93.676241,-93.665737,-93.664768,-93.663509,-93.664686,-93.661542,-93.660303,-93.665875,-93.66346,-93.663455,-93.663329,-93.646688,-93.644669,-93.6447083,-93.644511,-93.644698,-93.64476,-93.646289,-93.643215,-93.64281,-93.644426,-93.641322,-93.642359,-93.640497,-93.641156,-93.642663,-93.6440062,-93.640326,-93.645785,-93.643548,-93.643429,-93.640182,-93.628412,-93.6295,-93.597005,-93.626875,-93.62826,-93.625313,-93.629496,-93.628236,-93.626689,-93.613886,-93.615697,-93.615618,-93.606577,-93.606672,-93.607597,-93.607551,-93.606933,-93.60104,-93.647403,-93.650588,-93.650569,-93.650561,-93.651732,-93.650417,-93.644366,-93.646782,-93.646791,-93.646788,-93.646894,-93.646634,-93.647585,-93.647199,-93.662162,-93.653109,-93.652429,-93.608192,-93.6072813,-93.607395,-93.605355,-93.605354,-93.601507,-93.602094,-93.603422,-93.604923,-93.604644,-93.604521,-93.60678,-93.607114,-93.604858,-93.605119,-93.604146,-93.604858,-93.618182,-93.616439,-93.617432,-93.639067,-93.637566,-93.636378,-93.636052,-93.635967,-93.635935,-93.635932,-93.631604,-93.635716,-93.636949,-93.639293,-93.634231,-93.631572,-93.632571,-93.629337,-93.629174,-93.628607,-93.62877,-93.626337,-93.626477,-93.6220148,-93.625169,-93.63007,-93.628269,-93.638026,-93.637874,-93.6326645,-93.631515,-93.633002,-93.635486,-93.6353259,-93.636044,-93.635367,-93.630348,-93.632355,-93.630289,-93.631162,-93.62626,-93.625373,-93.627849,-93.624411,-93.627103,-93.627328,-93.629846,-93.629516,-93.628857,-93.624767,-93.658864,-93.658413,-93.658226,-93.6566537,-93.654774,-93.657984,-93.658788,-93.652334,-93.654831,-93.653452,-93.6531488,-93.643412,-93.643862,-93.657883,-93.653084,-93.650626,-93.6534099,-93.649714,-93.654138,-93.649525,-93.654124,-93.65414,-93.650617,-93.645774,-93.644411,-93.644306,-93.640287,-93.640227,-93.641321,-93.6419,-93.642261,-93.642241,-93.642116,-93.641518,-93.641522,-93.64458,-93.642588,-93.6436049,-93.640211,-93.642713,-93.642913,-93.641281,-93.64136,-93.641283,-93.641053,-93.657851,-93.6532,-93.650059,-93.653915,-93.655997,-93.655704,-93.653608,-93.653479,-93.642317,-93.64178,-93.641793,-93.643635,-93.643811,-93.650361,-93.64407,-93.643898,-93.642862,-93.642309,-93.642374,-93.64357,-93.643173,-93.643859,-93.643977,-93.692575,-93.691403,-93.691787,-93.691754,-93.690959,-93.691246,-93.689021,-93.687695,-93.686267,-93.68665,-93.681407,-93.675963,-93.677123,-93.675723,-93.677012,-93.671037,-93.672411,-93.670919,-93.669708,-93.669757,-93.671926,-93.660643,-93.653876,-93.653512,-93.650474,-93.650057,-93.647832,-93.646526,-93.644891,-93.644891,-93.644891,-93.644891,-93.645617,-93.643373,-93.648931,-93.649329,-93.649352,-93.649329,-93.658237,-93.633752,-93.6346,-93.634738,-93.633466,-93.630258,-93.633098,-93.627935,-93.625976,-93.626097,-93.626161,-93.627517,-93.628069,-93.6269,-93.626345,-93.624612,-93.629155,-93.623414,-93.623564,-93.622312,-93.630156,-93.626765,-93.627065,-93.6282949,-93.621621,-93.621362,-93.622017,-93.621399,-93.628568,-93.629256,-93.6270261,-93.624738,-93.616852,-93.615629,-93.6181153,-93.620355,-93.618389,-93.619334,-93.618625,-93.612886,-93.612566,-93.614251,-93.612462,-93.618319,-93.618593,-93.612464,-93.613491,-93.6136767,-93.615474,-93.61547,-93.612263,-93.620354,-93.617182,-93.6143498,-93.613965,-93.610772,-93.612239,-93.6116427,-93.612266,-93.614049,-93.612404,-93.613899,-93.608909,-93.608984,-93.607582,-93.60942,-93.610278,-93.610501,-93.607257,-93.606508,-93.605727,-93.609054,-93.608903,-93.608904,-93.606743,-93.609053,-93.609053,-93.609053,-93.606892,-93.606893,-93.606744,-93.603704,-93.605839,-93.604568,-93.6050674,-93.6195519,-93.617139,-93.618489,-93.619513,-93.615681,-93.618335,-93.615638,-93.614011,-93.614615,-93.613905,-93.607862,-93.608833,-93.610354,-93.608938,-93.607744,-93.606803,-93.613889,-93.615365,-93.6127958,-93.613763,-93.608901,-93.610438,-93.609665,-93.606195,-93.626022,-93.625576,-93.620433,-93.623642,-93.62454,-93.624553,-93.623481,-93.621484,-93.620411,-93.6243778,-93.622386,-93.625436,-93.621562,-93.622624,-93.626858,-93.626631,-93.6275675,-93.6273219,-93.628416,-93.6265888,-93.65565,-93.65815,-93.6578344,-93.6600396,-93.655885,-93.655786,-93.67555,-93.675399,-93.675408,-93.675606,-93.672278,-93.674385,-93.673601,-93.672645,-93.670204,-93.670514,-93.665569,-93.66563,-93.667081,-93.669544,-93.66412,-93.668392,-93.667082,-93.662039,-93.678165,-93.676352,-93.674402,-93.676681,-93.666035,-93.665153,-93.663475,-93.661562,-93.660664,-93.660227,-93.666095,-93.666096,-93.660187,-93.6627359,-93.658506,-93.685537,-93.683235,-93.68218,-93.689071,-93.690366,-93.691234,-93.688925,-93.688924,-93.689485,-93.689084,-93.688934,-93.68216,-93.681349,-93.682095,-93.683877,-93.680401,-93.691896,-93.688302,-93.690363,-93.692221,-93.690215,-93.68923,-93.691243,-93.688577,-93.686825,-93.687158,-93.683468,-93.683246,-93.681378,-93.687703,-93.686288,-93.685504,-93.685697,-93.685571,-93.683718,-93.679147,-93.681479,-93.679803,-93.682686,-93.683728,-93.683777,-93.687684,-93.687685,-93.687826,-93.684089,-93.684116,-93.684116,-93.685369,-93.683435,-93.684332,-93.683758,-93.678267,-93.676369,-93.676361,-93.675297,-93.675548,-93.672379,-93.671304,-93.671612,-93.670025,-93.669977,-93.669927,-93.670063,-93.670039,-93.669985,-93.669924,-93.661951,-93.659182,-93.659222,-93.660119,-93.663458,-93.661995,-93.656753,-93.659043,-93.655631,-93.648562,-93.651811,-93.651285,-93.646942,-93.646438,-93.648417,-93.648416,-93.64644,-93.6447118,-93.647027,-93.644307,-93.644258,-93.64048,-93.643443,-93.641155,-93.642319,-93.641002,-93.6416718,-93.640612,-93.642884,-93.644203,-93.639428,-93.643489,-93.628467,-93.62841,-93.628298,-93.629496,-93.626683,-93.625147,-93.615307,-93.61385,-93.615012,-93.604344,-93.616026,-93.616242,-93.599969,-93.606736,-93.603671,-93.603173,-93.601191,-93.603854,-93.644527,-93.649773,-93.646099,-93.646575,-93.646172,-93.646758,-93.64789,-93.610268,-93.608343,-93.608195,-93.608195,-93.607142,-93.604391,-93.603922,-93.604279,-93.601089,-93.603181,-93.603524,-93.600439,-93.6019,-93.601615,-93.603534,-93.6086884,-93.606842,-93.606847,-93.60019],3,null,null,{"interactive":true,"className":"","stroke":true,"color":"white","weight":5,"opacity":0.4,"fill":true,"fillColor":"white","fillOpacity":0.2},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null,null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641487],"lat":[42.060936,42.060977]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641488],"lat":[42.060936,42.061114]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641772],"lat":[42.060936,42.061149]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641781],"lat":[42.060936,42.061318]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.642447],"lat":[42.060936,42.061193]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]}],"limits":{"lat":[41.986509,42.063388],"lng":[-93.693153,-93.577427]}},"evals":[],"jsHooks":[]}</script>

---

# Resampling a 5-NN model <img src="images/parsnip.png" class="title-hex">

```r
knn_mod <- 
  nearest_neighbor(neighbors = 5) %>% 
  set_engine("kknn") %>% 
  set_mode("regression")

knn_wfl <- 
  workflow() %>% 
  add_model(knn_mod) %>% 
  add_formula(log10(Sale_Price) ~ Longitude + Latitude)
```

If we were compute this model with the training set:

```r
fit(knn_wfl, data = ames_train)
```

---

# Resampling a 5-NN model <img src="images/rsample.png" class="title-hex"><img src="images/purrr.png" class="title-hex"><img src="images/parsnip.png" class="title-hex"><img src="images/dplyr.png" class="title-hex">

Let's repeat that but using each of the 10 analysis sets:

```r
knn_res <- cv_splits %>%
  mutate(
    workflows = map(
      splits, 
      ~ fit(knn_wfl, data = analysis(.x))
    )
  )

knn_res
```

```
## #  10-fold cross-validation 
## # A tibble: 10 x 3
##    splits           id     workflows   
##  * <named list>     <chr>  <named list>
##  1 <split [2K/220]> Fold01 <workflow>  
##  2 <split [2K/220]> Fold02 <workflow>  
##  3 <split [2K/220]> Fold03 <workflow>  
##  4 <split [2K/220]> Fold04 <workflow>  
##  5 <split [2K/220]> Fold05 <workflow>  
##  6 <split [2K/220]> Fold06 <workflow>  
##  7 <split [2K/220]> Fold07 <workflow>  
##  8 <split [2K/220]> Fold08 <workflow>  
##  9 <split [2K/220]> Fold09 <workflow>  
## 10 <split [2K/219]> Fold10 <workflow>
```

---

# Predictions for each model <img src="images/rsample.png" class="title-hex"><img src="images/purrr.png" class="title-hex"><img src="images/parsnip.png" class="title-hex"><img src="images/dplyr.png" class="title-hex">

.pull-left-a-lot[

.font90[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)                          
```

]

]
.pull-right-a-little[

```
## # A tibble: 2,199 x 2
##    fold  .pred
##    <chr> <dbl>
##  1 1      5.36
##  2 1      5.28
##  3 1      5.27
##  4 1      5.06
##  5 1      5.37
##  6 1      5.30
##  7 1      5.14
##  8 1      5.08
##  9 1      5.10
## 10 1      5.24
## # … with 2,189 more rows
```
]

---

.pull-left-a-lot[

.font90[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)

prices <- map_dfr(
    knn_res$splits,  
    ~ assessment(.x) %>% select(Sale_Price)
  )  
```

]

]
.pull-right-a-little[

```
## # A tibble: 2,199 x 1
##    Sale_Price
##         <int>
##  1     216000
##  2     160000
##  3     216500
##  4     133000
##  5     254000
##  6     218500
##  7     108000
##  8     105000
##  9     132000
## 10     155000
## # … with 2,189 more rows
```
]

---

.pull-left-a-lot[

.font90[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)

prices <- map_dfr(
    knn_res$splits,  
    ~ assessment(.x) %>% select(Sale_Price)
  ) %>% 
* mutate(Sale_Price = log10(Sale_Price))
```

]

]
.pull-right-a-little[

```
## # A tibble: 2,199 x 1
##    Sale_Price
##         <dbl>
##  1       5.33
##  2       5.20
##  3       5.34
##  4       5.12
##  5       5.40
##  6       5.34
##  7       5.03
##  8       5.02
##  9       5.12
## 10       5.19
## # … with 2,189 more rows
```
]

---

# Compute RMSE for each model <img src="images/rsample.png" class="title-hex"><img src="images/purrr.png" class="title-hex"><img src="images/parsnip.png" class="title-hex"><img src="images/dplyr.png" class="title-hex">

.pull-left-a-lot[

.font90[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)

prices <- map_dfr(
    knn_res$splits,  
    ~ assessment(.x) %>% select(Sale_Price)
  ) %>% 
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates <- knn_pred %>%
  bind_cols(prices)
```

]

]
.pull-right-a-little[

```
## # A tibble: 2,199 x 3
##    fold  .pred Sale_Price
##    <chr> <dbl>      <dbl>
##  1 1      5.36       5.33
##  2 1      5.28       5.20
##  3 1      5.27       5.34
##  4 1      5.06       5.12
##  5 1      5.37       5.40
##  6 1      5.30       5.34
##  7 1      5.14       5.03
##  8 1      5.08       5.02
##  9 1      5.10       5.12
## 10 1      5.24       5.19
## # … with 2,189 more rows
```
]

---

.pull-left-a-lot[

.font90[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)

prices <- map_dfr(
    knn_res$splits,  
    ~ assessment(.x) %>% select(Sale_Price)
  ) %>% 
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates <- knn_pred %>%
  bind_cols(prices) %>% 
* group_by(fold)
```

]

]
.pull-right-a-little[

```
## # A tibble: 2,199 x 3
## # Groups:   fold [10]
##    fold  .pred Sale_Price
##    <chr> <dbl>      <dbl>
##  1 1      5.36       5.33
##  2 1      5.28       5.20
##  3 1      5.27       5.34
##  4 1      5.06       5.12
##  5 1      5.37       5.40
##  6 1      5.30       5.34
##  7 1      5.14       5.03
##  8 1      5.08       5.02
##  9 1      5.10       5.12
## 10 1      5.24       5.19
## # … with 2,189 more rows
```
]

---

.pull-left-a-lot[

.font90[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)

prices <- map_dfr(
    knn_res$splits,  
    ~ assessment(.x) %>% select(Sale_Price)
  ) %>% 
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates <- knn_pred %>%
  bind_cols(prices) %>% 
  group_by(fold) %>% 
* do(rmse = rmse(., Sale_Price, .pred))
```

]

]
.pull-right-a-little[

```
## Source: local data frame [10 x 2]
## Groups: <by row>
## 
## # A tibble: 10 x 2
##    fold  rmse            
##  * <chr> <list>          
##  1 1     <tibble [1 × 3]>
##  2 10    <tibble [1 × 3]>
##  3 2     <tibble [1 × 3]>
##  4 3     <tibble [1 × 3]>
##  5 4     <tibble [1 × 3]>
##  6 5     <tibble [1 × 3]>
##  7 6     <tibble [1 × 3]>
##  8 7     <tibble [1 × 3]>
##  9 8     <tibble [1 × 3]>
## 10 9     <tibble [1 × 3]>
```

```
## # A tibble: 1 x 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 rmse    standard       0.113
```
]

---

.pull-left-a-lot[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)

prices <- map_dfr(
    knn_res$splits,  
    ~ assessment(.x) %>% select(Sale_Price)
  ) %>% 
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates <- knn_pred %>%
  bind_cols(prices) %>% 
  group_by(fold) %>% 
  do(rmse = rmse(., Sale_Price, .pred)) %>% 
* unnest(cols = c(rmse))
```

]

.pull-right-a-little[

```
## # A tibble: 10 x 4
##    fold  .metric .estimator .estimate
##    <chr> <chr>   <chr>          <dbl>
##  1 1     rmse    standard      0.113 
##  2 10    rmse    standard      0.0874
##  3 2     rmse    standard      0.0895
##  4 3     rmse    standard      0.102 
##  5 4     rmse    standard      0.0888
##  6 5     rmse    standard      0.0911
##  7 6     rmse    standard      0.101 
##  8 7     rmse    standard      0.103 
##  9 8     rmse    standard      0.0971
## 10 9     rmse    standard      0.113
```
]

---

# Compute Overall RMSE estimate <img src="images/rsample.png" class="title-hex"><img src="images/purrr.png" class="title-hex"><img src="images/parsnip.png" class="title-hex"><img src="images/dplyr.png" class="title-hex">

.pull-left-a-lot[

```r
knn_pred <- map2_dfr(
  knn_res$workflows, 
  knn_res$splits,     
  ~ predict(.x, assessment(.y)),         
  .id = "fold"
)

prices <- map_dfr(
    knn_res$splits,  
    ~ assessment(.x) %>% select(Sale_Price)
  ) %>% 
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates <- knn_pred %>%
  bind_cols(prices) %>% 
  group_by(fold) %>% 
  do(rmse = rmse(., Sale_Price, .pred)) %>% 
  unnest(cols = c(rmse))

*mean(rmse_estimates$.estimate)
```

```
## [1] 0.09851353
```

]

---

# How you probably feel right now

---

# Resampling Notes

This is actually a _simple case_:

* There is no attached recipe to be prepared for each resample.
 
 * It assumes that 5 neighbors is optimal. 
 
Common Questions:

* Q: What happens to these 10 models? Is this some ensemble thing? 
 
   * A: They are only used for measuring performance so `/dev/null`.
 
 * Q: What were we measuring again?  
 
   * A: The resampling estimate is how we think a 5-NN model fit on the _entire data set_ would preform.

---
layout: false
class: middle, center

# Q: Do you actually expect us to
# do anything like this again?

---

---

# Please don't be mad about the next few slides

---

# Easy resampling using the {tune} package <img src="images/tune.png" class="title-hex">

There is a `fit_resamples()` function in the `tune` package that does all of this for you.

```r
library(tune)
easy_eval <- fit_resamples(knn_wfl, resamples = cv_splits, control = control_resamples(save_pred = TRUE))
```

This does the same process that we did manually except the resamples models are not saved (but you could save them).

.code90[

```r
easy_eval
```

```
## #  10-fold cross-validation 
## # A tibble: 10 x 5
##    splits           id     .metrics         .notes           .predictions      
##  * <list>           <chr>  <list>           <list>           <list>            
##  1 <split [2K/220]> Fold01 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  2 <split [2K/220]> Fold02 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  3 <split [2K/220]> Fold03 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  4 <split [2K/220]> Fold04 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  5 <split [2K/220]> Fold05 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  6 <split [2K/220]> Fold06 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  7 <split [2K/220]> Fold07 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  8 <split [2K/220]> Fold08 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
##  9 <split [2K/220]> Fold09 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [220 × 3]>
## 10 <split [2K/219]> Fold10 <tibble [2 × 3]> <tibble [0 × 1]> <tibble [219 × 3]>
```

]

---

# Getting the statistics and predictions <img src="images/tune.png" class="title-hex"><img src="images/dplyr.png" class="title-hex">

.pull-left[

```r
collect_predictions(easy_eval) %>% 
  arrange(.row) %>% 
  slice(1:5) 
```

```
## # A tibble: 5 x 4
##   id     .pred  .row `log10(Sale_Price)`
##   <chr>  <dbl> <int>               <dbl>
## 1 Fold10  5.22     1                5.24
## 2 Fold10  5.34     2                5.39
## 3 Fold10  5.28     3                5.28
## 4 Fold09  5.28     4                5.29
## 5 Fold04  5.43     5                5.33
```

```r
collect_metrics(easy_eval)
```

```
## # A tibble: 2 x 5
##   .metric .estimator   mean     n std_err
##   <chr>   <chr>       <dbl> <int>   <dbl>
## 1 rmse    standard   0.0985    10 0.00298
## 2 rsq     standard   0.698     10 0.0153
```

]

.pull-right[

```r
collect_metrics(easy_eval, summarize = FALSE) %>% 
  slice(1:10)
```

```
## # A tibble: 10 x 4
##    id     .metric .estimator .estimate
##    <chr>  <chr>   <chr>          <dbl>
##  1 Fold01 rmse    standard      0.113 
##  2 Fold01 rsq     standard      0.640 
##  3 Fold02 rmse    standard      0.0895
##  4 Fold02 rsq     standard      0.724 
##  5 Fold03 rmse    standard      0.102 
##  6 Fold03 rsq     standard      0.713 
##  7 Fold04 rmse    standard      0.0888
##  8 Fold04 rsq     standard      0.767 
##  9 Fold05 rmse    standard      0.0911
## 10 Fold05 rsq     standard      0.717
```

Note that the outcome data are already logged.

]

---
layout: false
class: inverse, middle, center

# Model Tuning

---

# Tuning Parameters

There are some models with parameters that _cannot be directly estimated from the data_.

For example:

* The number of neighbors in a K-NN models.

* The depth of a classification tree.

* The link function in a generalized linear model (e.g. logit, probit, etc).

* The covariance structure in a linear mixed model.

---

# Tuning Parameters and Overfitting

Overfitting occurs when a model inappropriately picks up on trends in the training set that do not generalize to new samples.

When this occurs, assessments of the model based on the training set can show good performance that does not reproduce in future samples.

For example, _K_ = 1 neighbors is much more likely to overfit the data than larger values since they average more values.

Also, how would you evaluate this model by re-predicting the training set? Those values would be optimistic since one of your neighbors is always you.

---

# Model Tuning

.pull-left[

Unsurprisingly, we will evaluate a tuning parameter by fitting a model on one set of data and assessing it with another.

_Grid search_ uses a pre-defined set of candidate tuning parameter values and evaluates their performance so that the best values can be used in the final model.

We'll use resampling to do this. If there are _B_ resamples and _C_ tuning parameter combinations, we end up fitting  `$B \times C$`  models (but these can be done in parallel).

]
.pull-right[

]

---

# Better API's using the tune package

`tune` has more general functions for tuning models. There are two main strategies used:

* Grid search (as shown above) where all of the candidate models are known at the start. We pick the best of these.

* Iterative search where each iteration finds novel tuning parameter values to evaluate.

Both have their advantages and disadvantages. At first, we will focus on grid search.

---

# Elements Required for Grid Search <img src="images/dials.png" class="title-hex">

.pull-left[

* A set of candidate values to evaluate.

* Measure of model performance.

* A resampling scheme to reliably estimate model performance.

We've discussed the last two. Now let's focus on the grid.

There are two main types of grids: regular and non-regular.

]

.pull-right[

]

---

# About Regular Grids

Usually combinatorial representation of vectors of tuning parameter values. Note that:

* The number of values don't have to be the same per parameter.

* The values can be regular on a transformed scale (e.g. log-10 for penalty).

* Quantitative and qualitative parameters can be combined.

* As the number of parameters increase, the curse of dimensionality kicks in.

* _Thought_ to be really inefficient but not in all cases (see the sub-model trick and `multi_predict()`).

* Bad when performance plateaus over a range of one or more parameters.

---

# Making Regular Grids <img src="images/dials.png" class="title-hex">

.pull-left[

You can use `expand.grid()` to make a data frame where parameters are in columns and candidate values are in rows.

`dials` has functions to create parameters and parameter sets. For example:

```r
penalty()
```

```
## Amount of Regularization  (quantitative)
## Transformer:  log-10 
## Range (transformed scale): [-10, 0]
```

```r
mixture()
```

```
## Proportion of lasso Penalty  (quantitative)
## Range: [0, 1]
```

```r
glmn_param <- parameters(penalty(), mixture())
```

]

.pull-right[

```r
glmn_param
```

```
## Collection of 2 parameters for tuning
## 
##       id parameter type object class
##  penalty        penalty    nparam[+]
##  mixture        mixture    nparam[+]
```

```r
glmn_grid <- 
  grid_regular(glmn_param, levels = c(10, 5))
glmn_grid %>% slice(1:4)
```

```
## # A tibble: 4 x 2
##         penalty mixture
## *         <dbl>   <dbl>
## 1 0.0000000001        0
## 2 0.00000000129       0
## 3 0.0000000167        0
## 4 0.000000215         0
```

Note that the grid for `penalty` is created in the log-10 space but the values in the data frame are in the original units.

]

---

# Non-Regular Grids <img src="images/dials.png" class="title-hex">

.pull-left[

There are two main methods that we have to make these:

* Random grids uniformly sample the parameter space (that might already be on a different scale).

* Space-filling designs (SFD) are based on statistical experimental design principles and try to keep candidate values away from one another while encompassing the entire parameter space.

There's no real downside to using the space-filling designs, so we will focus on these.

]

.pull-right[

The code is easy when using a parameter set:

```r
set.seed(7454)

glmn_sfd <- grid_max_entropy(glmn_param, size = 50)

glmn_sfd %>% slice(1:4)
```

```
## # A tibble: 4 x 2
##      penalty mixture
##        <dbl>   <dbl>
## 1 0.00000362   0.172
## 2 0.0102       0.414
## 3 0.00000372   0.346
## 4 0.000709     0.544
```

```r
# grid_latin_hypercube() can also be used
# grid_random() too
```

]

---

# Modifying Parameter Sets <img src="images/dials.png" class="title-hex">

.pull-left[

```r
# The names can be changed:
glmn_set <- parameters(lambda = penalty(), mixture())

# The ranges can also be set by their name:
glmn_set <- 
  update(glmn_set, lambda = penalty(c(-5, -1)))
```

<br>

```r
# Some parameters depend on data dimensions:
mtry()
```

```
## # Randomly Selected Predictors  (quantitative)
## Range: [1, ?]
```

```r
rf_set <- parameters(mtry(), trees())
```

]

.pull-right[

```r
rf_set
```

```
## Collection of 2 parameters for tuning
## 
##     id parameter type object class
##   mtry           mtry    nparam[?]
##  trees          trees    nparam[+]
## 
## Parameters needing finalization:
##    # Randomly Selected Predictors ('mtry')
## 
## See `?dials::finalize` or `?dials::update.parameters` for more information.
```

```r
# Sets the range of mtry to
# be the number of predictors
finalize(rf_set, mtcars %>% dplyr::select(-mpg))
```

```
## Collection of 2 parameters for tuning
## 
##     id parameter type object class
##   mtry           mtry    nparam[+]
##  trees          trees    nparam[+]
```

]

---

# Hands-On: K-NN Grids

Look at the help file `?nearest_neighbors` and find the names of the three tuning parameters.

Create a parameter set for these three, make at least one grid, and plot them.

---

# K-NN Tuning parameters <img src="images/dials.png" class="title-hex">

The `dist_power` parameter is related to the type of distance calculation. `dist_power = 2` is everyday Euclidean distance, `dist_power = 1` is Manhattan distance, and so on. Fractional values are acceptable.

`weight_func` determines how much influence neighbors have based on their distance:

---

# Tagging Tuning parameters <img src="images/parsnip.png" class="title-hex"><img src="images/dials.png" class="title-hex">

To tune the model, the first step is to _tag_ the parameters that will be optimized. `tune::tune()` can be used to do this:

```r
library(tune)

knn_mod <- 
  nearest_neighbor(neighbors = tune(), weight_func = tune()) %>% 
  set_engine("kknn") %>% 
  set_mode("regression")
```

`parameters()` can detect these arguments:

```r
parameters(knn_mod)
```

```
## Collection of 2 parameters for tuning
## 
##           id parameter type object class
##    neighbors      neighbors    nparam[+]
##  weight_func    weight_func    dparam[+]
```
 
---

# Tagging Tuning parameters <img src="images/parsnip.png" class="title-hex"><img src="images/dials.png" class="title-hex">

`tune()` has an optimal argument called `id` that can be used to name the parameters:

```r
nearest_neighbor(neighbors = tune("K"), weight_func = tune("weights")) %>% 
  set_engine("kknn") %>% 
  set_mode("regression") %>% 
  parameters()
```

```
## Collection of 2 parameters for tuning
## 
##       id parameter type object class
##        K      neighbors    nparam[+]
##  weights    weight_func    dparam[+]
```

This mainly comes in handy when the same parameter type shows up in two different places (an example is shown later).

Recipe steps can also have `tune()` in their arguments.

---

# Grid Search <img src="images/tune.png" class="title-hex"><img src="images/parsnip.png" class="title-hex"><img src="images/dials.png" class="title-hex">

Let's tune the model over a regular grid and optimize `neighbors` and `weight_func`. First we make the grid, then call `tune_grid()`:

.pull-left[

```r
set.seed(522)

knn_grid <- knn_mod %>% 
  parameters() %>% 
  grid_regular(levels = c(15, 5))

ctrl <- control_grid(verbose = TRUE)

knn_tune <- tune_grid(
  ames_rec, 
  model = knn_mod, 
  resamples = cv_splits, 
  grid = knn_grid, 
  control = ctrl
)
```

]

.pull-right[

We are optimizing 75 models over 10 resamples. The output looks like:

```
✓ Fold04: recipe
❯ Fold04: model 1/5
✓ Fold04: model 1/5
! Fold04: model 1/5 (predictions): some 'x' values beyond boundary knots may cause ill-conditioned bases
❯ Fold04: model 2/5
✓ Fold04: model 2/5
! Fold04: model 2/5 (predictions): some 'x' values beyond boundary knots may cause ill-conditioned bases
```

]

---

# Some Notes

* If we are evaluating 75 models, why does it say `model 1/5`?
 
 * The recipe is made _once_ per resample to avoid redundant computations. 
 
 * The warnings about "ill-conditioned bases" is related to using splines for longitude and latitude. Remember those outlying houses on Lincoln Highway? 
 
   * These are printed _as they happen_ and are labeled so that you know which sub-model had the issue.

* Unfortunately, these messages don't show up when parallel processing is used (more on that later). We are working on this but it is a tough problem that I've been trying to solve for about a decade.

---

# The Results  <img src="images/tune.png" class="title-hex"><img src="images/parsnip.png" class="title-hex"><img src="images/dials.png" class="title-hex">

A tibble is returned that looks like the <span class="note">rsample</span> object with an extra list column called `.metrics`:

.code90[

.pull-left[

```r
knn_tune
```

```
## #  10-fold cross-validation 
## # A tibble: 10 x 4
##    splits           id     .metrics           .notes          
##  * <list>           <chr>  <list>             <list>          
##  1 <split [2K/220]> Fold01 <tibble [150 × 5]> <tibble [0 × 1]>
##  2 <split [2K/220]> Fold02 <tibble [150 × 5]> <tibble [0 × 1]>
##  3 <split [2K/220]> Fold03 <tibble [150 × 5]> <tibble [0 × 1]>
##  4 <split [2K/220]> Fold04 <tibble [150 × 5]> <tibble [5 × 1]>
##  5 <split [2K/220]> Fold05 <tibble [150 × 5]> <tibble [0 × 1]>
##  6 <split [2K/220]> Fold06 <tibble [150 × 5]> <tibble [0 × 1]>
##  7 <split [2K/220]> Fold07 <tibble [150 × 5]> <tibble [5 × 1]>
##  8 <split [2K/220]> Fold08 <tibble [150 × 5]> <tibble [5 × 1]>
##  9 <split [2K/220]> Fold09 <tibble [150 × 5]> <tibble [5 × 1]>
## 10 <split [2K/219]> Fold10 <tibble [150 × 5]> <tibble [0 × 1]>
```

]

.pull-right[

```r
# results for the first fold:
knn_tune$.metrics[[1]]
```

```
## # A tibble: 150 x 5
##    neighbors weight_func .metric .estimator .estimate
##        <int> <chr>       <chr>   <chr>          <dbl>
##  1         1 biweight    rmse    standard      0.111 
##  2         2 biweight    rmse    standard      0.103 
##  3         3 biweight    rmse    standard      0.0982
##  4         4 biweight    rmse    standard      0.0962
##  5         5 biweight    rmse    standard      0.0950
##  6         6 biweight    rmse    standard      0.0940
##  7         7 biweight    rmse    standard      0.0925
##  8         8 biweight    rmse    standard      0.0923
##  9         9 biweight    rmse    standard      0.0926
## 10        10 biweight    rmse    standard      0.0929
## # … with 140 more rows
```

150 rows = 75 models x 2 metrics

]

---

# Resampled Performance Estimates  <img src="images/tune.png" class="title-hex"><img src="images/parsnip.png" class="title-hex"><img src="images/dplyr.png" class="title-hex"><img src="images/dials.png" class="title-hex">

To get the overall resampling estimate (averaged over folds) for each parameter combination:

```r
show_best(knn_tune, metric = "rmse", maximize = FALSE)
```

```
## # A tibble: 5 x 7
##   neighbors weight_func .metric .estimator   mean     n std_err
##       <int> <chr>       <chr>   <chr>       <dbl> <int>   <dbl>
## 1         8 triangular  rmse    standard   0.0816    10 0.00281
## 2         7 triangular  rmse    standard   0.0817    10 0.00278
## 3         6 triangular  rmse    standard   0.0817    10 0.00279
## 4         9 triangular  rmse    standard   0.0818    10 0.00283
## 5        11 triangular  rmse    standard   0.0819    10 0.00299
```