Navigating the source!

[gcliddellSoton]

I'm working on environmental signal processing and event detection and have started implementing lots of functions that are in this package for block-wise processing on super long 1D signals that don't fit in my available memory. I'd love to look at some of the implementations in this package for style inspiration, so that if they turn out well I could hand them over, but I actually can't find the file where the entropy functions are implemented (not a great start!). I can see them being imported, e.g. "from .core.information_theory import entropy" but there is no file "core/information_theory.py" in the repository anywhere. Am I being a complete idiot?

[Nikeshbajaj]

Hi @gcliddellSoton , good catch! Somehow not all the files were uploaded to github. I have uploaded the files on github now too, in case you like to use them. Also, when you install spkit, you get all the files on your local machine at lib/python3.9/site-packages/spkit/.

For block processing, I would suggest to use overlap-add method, that works well for most of processing techniques. This can be also parallarised (or use threading) to make them even faster.

I hope that helps!

[gcliddellSoton]

That does indeed help a lot. Thank you.

And pointing out that 'Overlap-add' is the correct terminology to search for is also very useful! I've just been using a time-domain threshold for event detection and the events are very short and rare, so no edge handling has really been necessary yet. It seems to me that the time-domain entropy calculations don't require overlaps as the histograms can be created iteratively first, I'm sure I could've implemented the binning better, but it seems to work!

[Nikeshbajaj]

Yes, Overlap-add is a common method used for window processing, to make sure no information is lost due to being near the edge of the window. It is primarly used for the reconstruction of the signal. In spkit, several methods using windowing approaches are implemented using overlap add. I am not sure, if you would really need this. Yes, time-domain entropy doesn't use overlap-add, as histogram calculation is quite fast. For bining, in spkit we have several methods and for entropy calculation, you could use any of these methods by passing to "bins"