tests of tutorials

[yogeshhk]

Unless already in place (or planned), is it a good idea to create tests out of workflows mentioned in the tutorials, just to ensure sanity of the tutorials themselves?

Like:
test_tutorial_stumpy_basics.py

import numpy as np
import numpy.testing as npt
import pandas as pd
import stumpy


steam_df = pd.read_csv("https://zenodo.org/record/4273921/files/STUMPY_Basics_steamgen.csv?download=1")
steam_m = 640
steam_mp = stumpy.stump(steam_df['steam flow'], steam_m)

def test_stump_steam_find_motif():
    min_value = steam_mp[:, 0].min()
    npt.assert_almost_equal(5.491619827769589, min_value) 

def test_stump_steam_find_anomalies():
    max_value = steam_mp[:, 0].max()
    npt.assert_almost_equal(23.476168367301977, max_value)  
    max_index = np.argwhere(steam_mp[:, 0] == steam_mp[:, 0].max()).flatten()[0]
    npt.assert_almost_equal(3864, max_index) 

[seanlaw]

@yogeshhk Thank you for the suggestion. This is not currently on our roadmap but I am certainly open to it. I have a few thoughts/concerns:

1. Have you discovered any notebooks that have bad/failed workflows? Regardless, those should be reported and fixed
2. I want to try and avoid having to extract code from the notebooks and then add them to a separate test file (and repeat this each time we update/modify an existing tutorial). This seems laborious and, possibly, more work than necessary. It would be great if extraction process is somehow automated so nobody has to think about it.
3. There are some notebooks that utilize a random seed (i.e., approximate matrix profile, fluss/floss) and I want to try and avoid setting random seeds in notebooks as this can be bad for novice users who are copying/pasting code from our notebooks.
4. I think adding a new workflow to Github Actions would be the right place to do this?

At the end of the end, the question isn't whether this is a good engineering practice. It definitely is! But we need to evaluate the cost of long term maintenance of these tests and whether the time spent there is better spent adding more tests to the existing code base or adding new features. In other words, given limited hours (or minutes) in a day, where do we want to provide value to the user?

Of course, we are always open to contributions!

[yogeshhk]

Thanks for letting me know the thought process and I agree. I am planning to study Matrix Profile as a subject more and start using Stumpy on various problem domains.