title: "cdata Transforms"

author: "Win-Vector LLC"

date: "12/26/2017"

output:

tufte::tufte_html: default

tufte::tufte_handout:

citation_package: natbib

latex_engine: xelatex

tufte::tufte_book:

citation_package: natbib

latex_engine: xelatex

link-citations: yes

Start up and show original `Keras` plot.

```{r setup}

Or with a bit more color:

```{r setup2}

First plot: reproduce the original with bulk-renaming of columns (via the new [`cdata::map_fieldsD()`](https://winvector.github.io/cdata/reference/map_fieldsD.html) function).

First we perform the equivalent of a "shred", "un-pivot", or "gather":

```{r firstplot}

Then we define a value mapping table to build the new

value key columns that we need.

```{r fp2}

We apply that key map and we are ready to plot:

```{r akm}

Second plot: the steps to get to [WVPlots::plot_Keras_fit_trajectory()](https://winvector.github.io/WVPlots/reference/plot_Keras_fit_trajectory.html). In particular show the structure of the control table (especially when applied to itself). The idea is: any in-table block-structure can be taken to any block-structure by moving through a very wide column as an intermediate (or dual form: through a very thin intermediate structure such as RDF-triples).

First let's take a look at our data.

```{r lineplot}

Let's concentrate on a single row (in this case the first row).

```{r lineplot1}

To create `ggplot2::geom_ribbon()` and `ggplot2::geom_segment()` we

need both the training and validation loss for a given epoch to be in the same row.

We also want the different performance metrics to be in different rows so we can

use `ggplot2::facet_wrap()`. That means we want the first row of data to look like the following sub-table:

```{r lineplot2}

What allows this motion is the `controlTable` which is essentially a before or after

diagram of the transform (depending on which direction you are going). I can not emphasize

enough the benefit of looking at the data and drawing out the transform on paper *before*

attempting any coding.

The control table is as follows:

```{r lpcct}

It can be applied to data, or to itself, in a forward or backward direction (depending if we use `cdata::rowrecs_to_blocks()` or `cdata::blocks_to_rowrecs()`).

```{r lpc}

Try this with numbers.

```{r proto}

We can now apply the transform to all the data, and produce the final plot.

```{r moveall}

All of the above is now wrapped in convenient function: [WVPlots::plot_Keras_fit_trajectory()](https://winvector.github.io/WVPlots/reference/plot_Keras_fit_trajectory.html).

All of the above is based on the second generation fluid data theory behind the `cdata` package. The [first generation of the theory](http://winvector.github.io/FluidData/RowsAndColumns.html) was about establishing and maintaining invariant that make data transform reversible and commutative. The [second generation of the theory](http://winvector.github.io/FluidData/ArbitraryTransform.html) is about transforms beyond pivot/un-pivot (moving sets of values in unison).

A quick list of references:

* [Coordinatized Data: A Fluid Data Specification](http://winvector.github.io/FluidData/RowsAndColumns.html)

* [Arbitrary Data Transforms Using cdata](http://winvector.github.io/FluidData/ArbitraryTransform.html)

* [Data Wrangling at Scale](http://winvector.github.io/FluidData/DataWranglingAtScale.html)

* [Big Data Transforms](http://winvector.github.io/FluidData/BigDataTransforms.html)

* [cdata Transforms](http://winvector.github.io/FluidData/PlotExample/PlotExample.html) (*this article*)

* [Plotting Keras Performance Trajectories](http://winvector.github.io/FluidData/PlotExample/KerasPerfPlot.html)

* [Plotting xgboost Performance Trajectories](http://winvector.github.io/FluidData/PlotExample/xgboostPerfPlot.html)