Data collection

[ijlyttle]

The goal here is to take a spec, go through it to find all the datasets, then to collect the datasets at the top of the spec, give them names, then refer to those names in the spec, where we found the data.

First idea:

vlc_collect_data <- function(spec) {

  # pull all the datasets out of a spec, rationalize
  datasets <- vlc_get_data(spec)

  # replace the datasets in a spec
  spec <- vlc_reset_data(spec, datasets)

  spec
}

Helper functions to be described below.

• vlc_get_data()
• vlc_reset_data()
• vlc_collect_data()
• vlc_describe_data()

[ijlyttle]

I think that there should be a couple of helper functions, perhaps internal, perhaps not exported:

This function would get all the data from a spec, then propose names. Here, values name, hash, and value are unique within each variable.

vlc_get_data <- function(spec) {
  # Return an unnamed list where each element is a dataset with elements:
  #   - `name`: something like `"data-00"`
  #   - `hash`: md5 hash of `value`
  #   - `value`: the value of the particular `data` element in the spec
  # 
  # If this were transposed, it would be data-frame with a list-column for `value`.
  # - perhaps it should be transposed 

  datasets
}

This function would replace all the data in a spec with the data from datasets

vlc_reset_data <- function(spec, datasets) {
  # Replace the datasets with new named datasets
  # - be mindful of renaming already-named datasets so that 
  #   the new name follows

  spec
}

This function would be used to get a sense of all the datasets, it would be exported.

vlc_describe_data <- function(spec) {
  # Return a data frame with variables:
  #   - `name` of the data set
  #   - `type` of the data set, e.g. `"url"`, `"values"`
  #   - `sample` (string) either the value of the URL or JSON serialization of the first row
  
  description
}