v2.0.0 (#79)

* Handling of null datetimes in pandas dataframes, pandas > 1.0 now default for support

* removal of error trap

* Minor change to structure of dt_convert

* unneccesary conversion to timestamp

* wording update

* addition of missing xval folder from releases

* updated ksdistrib to correspond with scipy

* sanity check for scipy version within select.q added

* new grid/random/sobol searching methods

* sobol requirements

* requirement updates for sobol

* check for number of random hps

* sobol check fix

* gs/rs tests

* addition of 1st pass graph/pipeline code

* initial pass at graphing tests

* Increased testing, covering base functionality

* update to account for testing and packaging of graph functionality

* updated tab2df to account for list of char being passed

* addition of most basic modules to ml toolkit (loading/saving data)

* addition of loading and saving modules to ml graph library (most basic generally useful module)

* update to graph functionality to handle generic dictionaries

* minor update to graph.t to increase converage and explicitly test the output error not just that function errors

* addition of node update failing functions

* added labelencode function and tests to preproc

* update to labelencode functionality and addition of application function for this

* addition of time series and optimization functionality for oct 7th release

* update to tests to correctly load data

* update to README to add in reference to optimization, graphing and time series

* added README for timeseries

* changed time series to lower case

* first pass at optimization tests

* addition of datasets for testing

* update to test data path

* update to tests to handle discrepencies between q versions

* update to optimize test and addition of dictCheck tests

* revert of prediction functionality as tests are already covered

* removal of sobol-seq from from requirements due to lack of conda install and issues with using this and conda

* reintroduction of requirements file

* trigger build

* reordering of requirements to trigger build

* stationality -> stationarity as this is the correct term

* renaming of dataset for stationarity

* movement from conda to pip install with docker

* fixed windowed feat function

* full coverage of failing tests for graph library

* updated appveyor files

* addition of width parameter to autocorrelation plot

* addition of timeseries and optimization tests

* included optimization files and fixed cluster test

* added saved input files for optim func

* restructure

* restructure and addition of windows data

* restructure and addition of windows data

* separate windows and linux folders

* updated optim tests for windows/linux

Co-authored-by: dmorgankx <44678213+dmorgankx@users.noreply.github.com>
Co-authored-by: Deanna Morgan <dmorgan1@kx.com>
Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com>
Co-authored-by: Dianeod <dodonoghue@kx.com>

Author: 3 people

.travis.yml

@@ -36,11 +36,11 @@ beforescript:
 script:
 - (cd clust && make && make install && make clean)
 - echo "Preparing version $TRAVIS_BRANCH-$TRAVIS_COMMIT"
-- tar czf ml_$TRAVIS_OS_NAME-$TRAVIS_BRANCH.tgz *.q fresh/ xval/ util/ clust/ requirements.txt  LICENSE README.md
+- tar czf ml_$TRAVIS_OS_NAME-$TRAVIS_BRANCH.tgz *.q fresh/ xval/ util/ clust/ graph/ timeseries/ optimize/ requirements.txt  LICENSE README.md
 - echo "Packaged as ml_$TRAVIS_OS_NAME-$TRAVIS_BRANCH.zip"
 -  if [[ "x$QLIC_KC" != "x" ]]; then
     curl -fsSL -o test.q https://github.com/KxSystems/embedpy/raw/master/test.q;
-    q test.q fresh/tests/ util/tests/ xval/tests clust/tests/ -q;
+    q test.q fresh/tests/ util/tests/ xval/tests clust/tests/ graph/tests/ timeseries/tests/ optimize/tests/ -q;
 
   else
     echo No kdb+, no tests;

README.md

@@ -7,9 +7,12 @@ The machine learning toolkit is at the core of kdb+/q centered machine learning
 *  An implementation of the FRESH (FeatuRe Extraction and Scalable Hypothesis testing) algorithm for use in the extraction of features from time series data and the reduction in the number of features through statistical testing. 
 *  Cross validation and grid-search functions allowing for testing of the stability of models to changes in the volume of data or the specific subsets of data used in training.
 *  Clustering algorithms used to group data points and to identify patterns in their distributions. The algorithms make use of a k-dimensional tree to store points and scoring functions to analyze how well they performed.
+*  Statistical time series models and feature extraction techniques used for the application of machine learning to time series problems. These models allow for the forecasting of the future behaviour of a system under various conditions.
+*  Numerical optimization techniques used for calculating the optimal parameters for an objective function.
+*  A graphing and pipeline library for the creation of modularized executable workflow based on a structure described by a mathematical directed graph.
 *  Utility functions relating to areas including statistical analysis, data preprocessing and array manipulation.
 
-These sections are explained in greater depth within the [FRESH](https://code.kx.com/v2/ml/toolkit/fresh/), [Cross Validation](https://code.kx.com/v2/ml/toolkit/xval), [Clustering](https://code.kx.com/v2/ml/toolkit/clustering/algos/) and [Utilities](https://code.kx.com/v2/ml/toolkit/utilities/metric) documentation.
+These sections are explained in greater depth within the [FRESH](https://code.kx.com/v2/ml/toolkit/fresh/), [cross validation](https://code.kx.com/v2/ml/toolkit/xval), [clustering](https://code.kx.com/v2/ml/toolkit/clustering/algos/), [time series](https://code.kx.com/v2/ml/toolkit/timeseries), [optimization](https://code.kx.com/v2/ml/toolkit/optimize/), [graph/pipeline](https://code.kx.com/v2/ml/toolkit/graph) and [utilities](https://code.kx.com/v2/ml/toolkit/utilities/metric) documentation.
 
 ## Requirements
 
@@ -27,6 +30,7 @@ or via conda:
 conda install --file requirements.txt
 ```
 
+
 ## Installation
 
 Place the `ml` library in `$QHOME` and load into a q instance using `ml/ml.q`
@@ -46,6 +50,7 @@ Examples showing implementations of several components of this toolkit can be fo
 *  Cross validation and grid search capabilities
 *  Results Scoring functionality
 *  Clustering methods applied to datasets
+*  Time series modeling examples
 
 ## Documentation
 

build/buildscript.bat

@@ -4,27 +4,20 @@ if "%APPVEYOR_REPO_TAG%"=="true" (
  set ML_VERSION=%APPVEYOR_REPO_BRANCH%_%APPVEYOR_REPO_COMMIT%
 )
 set PATH=C:\Perl;%PATH%
-perl -p -i.bak -e s/TOOLKITVERSION/`\$\"%ML_VERSION%\"/g ml.q
+perl -p -i.bak -e s/MLVERSION/`\$\"%ML_VERSION%\"/g ml.q
+
 
 if not defined QLIC_KC (
  goto :nokdb
 )
-call "build\getkdb.bat" || goto :error
 
-set PATH=C:\Miniconda3-x64;C:\Miniconda3-x64\Scripts;%PATH%
-mkdir embedpy
-cd embedpy
-echo getembedpy"latest" | q ..\build\getembedpy.q -q || goto :error
-cd ..
-echo p)print('embedpy runs') | q -q || goto :error
 
-cd clust/build
-call "build.bat" 2017
-cd ../..
+set PATH=C:\Miniconda3-x64;C:\Miniconda3-x64\Scripts;%PATH%
+conda config --set always_yes yes --set changeps1 no
+call "build\getkdb.bat" || goto :error
 
 exit /b 0
 
-
 :error
 echo failed with error 
 exit /b 

build/getembedpy.q

@@ -1,9 +1,8 @@
-curl -fsSOJL %W64%
+conda install -c kx embedPy
 mkdir q
-7z x w64.zip -oq
+xcopy /E C:\Miniconda3-x64\q q
 echo|set /P =%QLIC_KC% >q\kc.lic.enc
 certutil -decode q\kc.lic.enc q\kc.lic
 set QHOME=%cd%\q
 set PATH=%QHOME%\w64;%PATH%
-where q
-echo .z.K | q -q
+echo show .z.K;show .z.k;exit 0 | q -q

build/getkdb.bat

@@ -1,3 +1,3 @@
-7z a ml_windows-%ML_VERSION%.zip *.q fresh util xval clust requirements.txt LICENSE README.md
+7z a ml_windows-%ML_VERSION%.zip *.q fresh util xval clust graph timeseries optimize requirements.txt LICENSE README.md
 appveyor PushArtifact ml_windows-%ML_VERSION%.zip
 exit /b 0

build/package.bat

@@ -2,5 +2,5 @@ if defined QLIC_KC (
         pip -q install -r requirements.txt
 	echo getting test.q from embedpy
         curl -fsSL -o test.q https://github.com/KxSystems/embedpy/raw/master/test.q
-        q test.q fresh/tests/ util/tests/ xval/tests/ clust/tests/ -q
+        q test.q fresh/tests/ util/tests/ xval/tests/ clust/tests/ graph/tests/ timeseries/tests/ optimize/tests/ -q
 )

build/test.bat

@@ -28,7 +28,7 @@ info:.ml.clust.i.apinit[d1;`e2dist;max]
 .ml.clust.kd.newtree[d2;3][`parent]~0N 0 0
 .ml.clust.kd.newtree[d2;3][`idxs]~(0#0;til 5;5+til 5)
 .ml.clust.kd.nn[tree;d1;`edist;0;d2[;2]][`closestPoint]~2
-.ml.clust.kd.nn[tree;d1;`mdist;1 2 3 4;d1[;1]][`closestPoint`closestDist]~(0;2f)
+all .ml.clust.kd.nn[tree;d1;`mdist;1 2 3 4;d1[;1]][`closestPoint`closestDist]=(0;2f)
 .ml.clust.kd.nn[tree;d1;`mdist;1;7 9f][`closestPoint`closestDist]~(4;8f)
 .ml.clust.kd.nn[tree2;d2;`edist;1 2 3;d1[;1]][`closestPoint]~0
 .ml.clust.kd.nn[tree2;d2;`edist;1 5 2;d1[;3]][`closestPoint]~3

clust/tests/util.t

@@ -13,6 +13,9 @@ COPY fresh /opt/kx/ml/fresh
 COPY util /opt/kx/ml/util
 COPY xval /opt/kx/ml/xval
 COPY clust /opt/kx/ml/clust
+COPY graph /opt/kx/ml/graph
+COPY timeseries /opt/kx/ml/timeseries
+COPY optimize /opt/kx/ml/optimize
 
 ARG VCS_REF=dev
 ARG BUILD_DATE=dev
@@ -32,13 +35,15 @@ LABEL	org.label-schema.schema-version="0.1" \
 RUN chown -R kx:kx /opt/kx/ml 
 RUN mkdir /opt/kx/q/ml
 RUN find /opt/kx/ml -maxdepth 1 -type f -name '*.q' | xargs ln -s -t /opt/kx/q/ml \
-        && ln -s -t /opt/kx/q/ml /opt/kx/ml/fresh /opt/kx/ml/utils /opt/kx/ml/xval /opt/kx/ml/clust
+        && ln -s -t /opt/kx/q/ml /opt/kx/ml/fresh /opt/kx/ml/utils /opt/kx/ml/xval /opt/kx/ml/clust \
+        /opt/kx/ml/graph /opt/kx/ml/timeseries /opt/kx/ml/optimize
 
 USER kx
 
 RUN . /opt/conda/etc/profile.d/conda.sh \
         && conda activate kx \
-        && conda install -c conda-forge --file /opt/kx/ml/requirements.txt \
+        && pip install pip==9.0.1 \
+        && pip install -r /opt/kx/ml/requirements.txt \
         && conda clean -y --all 
 
 USER root

docker/Dockerfile

@@ -0,0 +1,36 @@
+# Graphing and Pipeline interface
+
+The functionality contained in this folder surrounds the implementation of a graph and pipeline execution structural form of kdb+. This functionality is intended to provide a common structural template and execution mechanism for complex code bases require ease of modification which is common within machine learning use cases.
+
+## Functionality
+
+Within this folder are two scripts that contains the entirity of this graph and pipeline functionality. These scripts are:
+
+1. graph.q: This contains all functionality required for the creation, deletion and update of nodes and edges within the graph structure.
+2. pipeline.q: This contains functionality for both the compilation and execution of a user generated graph.
+
+## Requirements
+
+- kdb+ > 3
+
+## Installation
+
+Place the `ml` library in `$QHOME` and load into a q instance using `ml/ml.q`
+
+### Load
+
+The following will load the graphing and pipeline functionality into the `.ml` namespace  
+```q
+q)\l ml/ml.q
+q).ml.loadfile`:graph/init.q
+```
+
+## Documentation
+
+Documentation is available on the [Graph](https://code.kx.com/q/ml/toolkit/graph/) homepage.
+
+## Status
+
+The graph-pipeline library is still in development and is available here as a beta release. Further functionality and improvements will be made to the library in the coming months.
+
+If you have any issues, questions or suggestions, please write to ai@kx.com.

graph/README.md

@@ -0,0 +1,86 @@
+\d .ml
+
+createGraph:{[]
+  nodes:1!enlist`nodeId``function`inputs`outputs!(`;::;::;::;::);
+  edges:2!enlist`dstNode`dstName`srcNode`srcName`valid!(`;`;`;`;0b);
+  `nodes`edges!(nodes;edges)}
+
+addNode:{[graph;nodeId;node]
+  node,:(1#`)!1#(::);
+  if[nodeId in exec nodeId from graph`nodes;'"invalid nodeId"];
+  if[not``function`inputs`outputs~asc key node;'"invalid node"];
+  if[(::)~node`inputs;node[`inputs]:(0#`)!""];
+  if[-10h=type node`inputs;node[`inputs]:(1#`input)!enlist node`inputs];
+  if[99h<>type node`inputs;'"invalid inputs"];
+  if[-10h=type node`outputs;
+    node[`outputs]:(1#`output)!enlist node`outputs;
+    node[`function]:((1#`output)!enlist@)node[`function]::;
+  ];
+  if[99h<>type node`outputs;'"invalid outputs"];
+  graph:@[graph;`nodes;,;update nodeId from node];
+  edges:flip`dstNode`dstName`srcNode`srcName`valid!(nodeId;key node`inputs;`;`;0b);
+  graph:@[graph;`edges;,;edges];
+  graph}
+
+updNode:{[graph;nodeId;node]
+  node,:(1#`)!1#(::);
+  if[not nodeId in 1_exec nodeId from graph`nodes;'"invalid nodeId"];
+  if[count key[node]except``function`inputs`outputs;'"invalid node"];
+  oldnode:graph[`nodes]nodeId;
+  if[`inputs in key node;
+    if[(::)~node`inputs;node[`inputs]:(0#`)!""];
+    if[-10h=type node`inputs;node[`inputs]:(1#`input)!enlist node`inputs];
+    if[99h<>type node`inputs;'"invalid inputs"];
+    inputEdges:select from graph[`edges]where dstNode=nodeId,dstName in key oldnode`inputs;
+    graph:@[graph;`edges;key[inputEdges]_];
+    inputEdges:flip[`dstNode`dstName!(nodeId;key node`inputs)]#inputEdges;
+    graph:@[graph;`edges;,;inputEdges];
+    inputEdges:select from inputEdges where not null srcNode;
+    graph:{[graph;edge]connectEdge[graph]. edge`srcNode`srcName`dstNode`dstName}/[graph;0!inputEdges];
+  ];
+  if[`outputs in key node;
+    if[-10h=type node`outputs;
+      node[`outputs]:(1#`output)!enlist node`outputs;
+    ];
+    if[99h<>type node`outputs;'"invalid outputs"];
+    outputEdges:select from graph[`edges]where srcNode=nodeId,srcName in key oldnode`outputs;
+    graph:@[graph;`edges;key[outputEdges]_];
+    outputEdges:select from outputEdges where srcName in key node`outputs;
+    graph:@[graph;`edges;,;outputEdges];
+    outputEdges:select srcNode,srcName,dstNode,dstName from outputEdges;
+    graph:{[graph;edge]connectEdge[graph]. edge`srcNode`srcName`dstNode`dstName}/[graph;0!outputEdges];
+  ];
+  if[`function in key node;
+    if[(1#`output)~key graph[`nodes;nodeId]`outputs;node[`function]:((1#`output)!enlist@)node[`function]::];
+  ];
+  graph:@[graph;`nodes;,;update nodeId from node];
+  graph}
+
+delNode:{[graph;nodeId]
+  if[not nodeId in 1_exec nodeId from graph`nodes;'"invalid nodeId"];
+  graph:@[graph;`nodes;_;nodeId];
+  inputEdges:select from graph[`edges]where dstNode=nodeId;
+  graph:@[graph;`edges;key[inputEdges]_];
+  outputEdges:select from graph[`edges]where srcNode=nodeId;
+  graph:@[graph;`edges;,;update srcNode:`,srcName:`,valid:0b from outputEdges];
+  graph}
+
+addCfg:{[graph;nodeId;cfg]addNode[graph;nodeId]``function`inputs`outputs!(::;@[;cfg];::;"!")}
+updCfg:{[graph;nodeId;cfg]updNode[graph;nodeId](1#`function)!enlist cfg}
+delCfg:delNode
+
+connectEdge:{[graph;srcNode;srcName;dstNode;dstName]
+  if[99h<>type srcOutputs:graph[`nodes;srcNode;`outputs];'"invalid srcNode"];
+  if[99h<>type dstInputs:graph[`nodes;dstNode;`inputs];'"invalid dstNode"];
+  if[not srcName in key srcOutputs;'"invalid srcName"];
+  if[not dstName in key dstInputs;'"invalid dstName"];
+  edge:(1#`valid)!1#srcOutputs[srcName]~dstInputs[dstName];
+  graph:@[graph;`edges;,;update dstNode,dstName,srcNode,srcName from edge];
+  graph}
+
+disconnectEdge:{[graph;dstNode;dstName]
+  if[not(dstNode;dstName)in key graph`edges;'"invalid edge"];
+  edge:(1#`valid)!1#0b;
+  graph:@[graph;`edges;,;update dstNode,dstName,srcName:`,srcNode:` from edge];
+  graph}
+

graph/graph.q

@@ -0,0 +1,4 @@
+.ml.loadfile`:graph/graph.q
+.ml.loadfile`:graph/pipeline.q
+.ml.loadfile`:graph/modules/saving.q
+.ml.loadfile`:graph/modules/loading.q

graph/init.q

@@ -0,0 +1,27 @@
+\d .ml
+  
+i.loadfname:{[cfg]
+  file:hsym`$$[`dir in key cfg;cfg`key;"."],"/",cfg fname;
+  if[()~key file;'"file does not exist"];
+  file}
+
+i.loadfunc.splay:loadfunc.binary:{[cfg]get i.loadfname cfg}
+i.loadfunc.csv:{[cfg](cfg`schema;enlist cfg`separator)0: i.loadfname cfg}
+i.loadfunc.json:{[cfg].j.k first read0 i.loadfname cfg}
+i.loadfunc.hdf5:{[cfg]
+  if[not`hdf5 in key`;@[system;"l hdf5.q";{'"unable to load hdf5 lib"}]];
+  if[not .hdf5.ishdf5 fname:i.loadfname cfg;'"file is not an hdf5 file"];
+  if[not .hdf5.isObject[fpath;cfg`dname];'"hdf5 dataset does not exist"];
+  .hdf5.readData[fpath;cfg`dname]}
+i.loadfunc.ipc:{[cfg]
+  h:@[hopen;cfg`prt;{'"error opening connection"}];
+  ret:@[h;cfg`select;{'"error executing query"}];
+  @[hclose;h;{}];
+  ret}
+i.loadfunc.process:{[cfg]if[not `data in key cfg;'"Data to be used must be defined"];cfg[`data]}
+
+i.loaddset:{[cfg]
+  if[null func:i.loadfunc cfg`typ;'"dataset type not supported"];
+  func cfg}
+
+loaddset:`function`inputs`outputs!(i.loaddset;"!";"+")

graph/modules/loading.q

@@ -0,0 +1,29 @@
+\d .ml
+  
+i.savefname:{[cfg]
+  file:hsym`$$[`dir in key cfg;cfg`key;"."],"/",cfg fname;
+  if[not ()~key file;'"file exists"];
+  file}
+
+i.savedset.txt:{[cfg;dset]i.savefname[cfg]0:.h.tx[cfg`typ;dset];}
+i.savedset[`csv`xml`xls]:i.savedset.txt
+i.savedset.binary:{[cfg;dset]i.savefname[cfg]set dset;}
+i.savedset.json:{[cfg;dset]
+  h:hopen i.savefname cfg;
+  h @[.j.j;dset;{'"error converting to json"}];
+  hclose h;}
+i.savedset.hdf5:{[cfg;dset]
+  if[not`hdf5 in key`;@[system;"l hdf5.q";{'"unable to load hdf5 lib"}]];
+  .hdf5.createFile fname:i.savefname cfg;
+  .hdf5.writeData[fname;cfg`dname;dset];
+  }
+i.savedset.splay:{[cfg;dset]
+  dname:first` vs fname:i.savefname cfg;
+  fname:` sv fname,`;
+  fname set .Q.en[dname]dset;}
+
+i.savefunc:{[cfg;dset]
+  if[null func:i.savedset cfg`typ;'"dataset type not supported"];
+  func dset}
+
+savedset:`function`inputs`outputs!(i.savefunc;`cfg`dset!"!+";" ")

graph/modules/saving.q

@@ -0,0 +1,51 @@
+\d .ml
+
+// Execution of a pipeline will not default to enter q debug mode but should be possible to overwrite
+graphDebug:0b
+updDebug:{[x]graphDebug::not graphDebug}
+
+createPipeline:{[graph]
+  if[not all exec 1_valid from graph`edges;'"disconnected edges"];
+  outputs:ungroup select srcNode:nodeId,srcName:key each outputs from 1_graph`nodes;
+  endpoints:exec distinct srcNode from outputs except select srcNode,srcName from graph`edges;
+  optimalpath:distinct raze paths idesc count each paths:i.getOptimalPath[graph]each endpoints;
+  pipeline:([]nodeId:optimalpath)#graph`nodes;
+  nodeinputs:key each exec inputs from pipeline;
+  pipeline:update inputs:count[i]#enlist(1#`)!1#(::),outputtypes:outputs,inputorder:nodeinputs from pipeline;
+  pipeline:select nodeId,complete:0b,error:`,function,inputs,outputs:inputs,outputtypes,inputorder from pipeline;
+  pipeline:pipeline lj select outputmap:([]srcName;dstNode;dstName)by nodeId:srcNode from graph`edges;
+  1!pipeline}
+
+execPipeline:{[pipeline]i.execCheck i.execNext/pipeline}
+
+
+// Pipeline creation utilities
+i.getDeps:{[graph;node]exec distinct srcNode from graph[`edges]where dstNode=node}
+i.getAllDeps:{[graph;node]$[count depNodes:i.getDeps[graph]node;distinct node,raze .z.s[graph]each depNodes;node]}
+i.getAllPaths:{[graph;node]$[count depNodes:i.getDeps[graph]node;node,/:raze .z.s[graph]each depNodes;raze node]}
+i.getLongestPath:{[graph;node]paths first idesc count each paths:reverse each i.getAllPaths[graph;node]}
+i.getOptimalPath:{[graph;node]distinct raze reverse each i.getAllDeps[graph]each i.getLongestPath[graph;node]}
+
+i.execNext:{[pipeline]
+  node:first 0!select from pipeline where not complete;
+  -1"Executing node: ",string node`nodeId;
+  if[not count inputs:node[`inputs]node[`inputorder];inputs:1#(::)];
+  res:`complete`error`outputs!$[graphDebug;
+      .[(1b;`;)node[`function]::;inputs];
+      .[(1b;`;)node[`function]::;inputs;{[err](0b;`$err;::)}]
+  ];
+  / compare outputs to outputtypes ?
+  if[not null res`error;-2"Error: ",string res`error];
+  if[res`complete;
+    res[`inputs]:(1#`)!1#(::);
+    outputmap:update data:res[`outputs]srcName from node`outputmap;
+    res[`outputs]:((1#`)!1#(::)),(exec distinct srcName from outputmap)_ res`outputs;
+    pipeline:{[pipeline;map]pipeline[map`dstNode;`inputs;map`dstName]:map`data;pipeline}/[pipeline;outputmap];
+  ];
+  pipeline,:update nodeId:node`nodeId from res;
+  pipeline}
+
+i.execCheck:{[pipeline]
+  if[any not null exec error from pipeline;:0b];
+  if[all exec complete from pipeline;:0b];
+  1b}