How Can I Cache STUMPY Functions and Reuse Them Between Python Sessions?

[seanlaw]

@damianzd1 had asked:

I've been trying but without success to cache stumpy.stump function so I can reuse it between scripts. In my particular case I am trying to build a docker container for AWS Lambda to be used as AWS API Gateway endpoint. The problem is that compiling stumpy.stump takes more time that AWS API Gateway allows a request to hang, so the call always fails. Can it be cached and reused between python sessions?

[seanlaw]

@damianzd1 Thank you for your question and welcome to the STUMPY community. This topic has been covered in more detail here and here are a few key points:

1. Caching functions ahead of time (AOT) is was added recently and currently only available in the development version of STUMPY and so you will need to install STUMPY from source. In other words caching functions is NOT available before STUMPY version 1.12.0! Note that this is still purely experimental so please use at your own risk and feel free to post any issues.
2. After you've installed the development version (and based on the link above), you should be able to cache STUMPY functions with:

import numpy as np
from stumpy import cache
import stumpy
import time

cache._enable()

T = np.random.rand(1000)
m = 50

start = time.time()
stumpy.stump(T, m)
print(time.time() - start)

start = time.time()
stumpy.stump(T, m)
print(time.time() - start)

You can also get a list of the cached functions via:

from stumpy import cache

for fname in cache._get_cache():
    print(fname)

and the cached functions (stored in site-packages/stumpy/__pycache__) can be cleared via:

from stumpy import cache

cache._clear()

Note that:

1. njit functions are only cached after being called once along a given execution path (i.e., all sub-functions are also cached)
2. even when all of the functions are cached, calling those functions for the first time still appears to incur a numba start-up/warm-up time (around 0.09 seconds on my M1 Macbook Air) and the second call to the function will still be much, much faster

Please feel free to ask any questions and provide any feedback. Again, this is purely experimental and should not be relied upon but, if it works, it's currently straightforward enough that I likely won't remove it (no guarantees though!). Use it at your own risk!

[seanlaw]

Based on this example, it has been discovered that the current implementation of the cache._clear() function may be insufficient and numba is still caching functions elsewhere beyond the usual places.

[seanlaw]

Based on #1048 (comment), it has been discovered that the current implementation of the cache._clear() function may be insufficient and numba is still caching functions elsewhere beyond the usual places.

A follow up question was posted was posted to the numba discourse channel

[seanlaw]

So, after enabling the cache followed by clearing the cache, numba will no longer create a cache unless the functions are explicitly recompiled:

from numba import njit
from os import listdir
import pathlib

PYCACHEDIR = '__pycache__'

@njit
def add(a, b):
    return a + b


if __name__ == "__main__":
    add.enable_caching()
    add(4, 5)
    print(listdir(PYCACHEDIR))
    [f.unlink() for f in pathlib.Path(PYCACHEDIR).glob("*nb*") if f.is_file()]
    print(listdir(PYCACHEDIR))
    add.recompile()
    add(4, 5)
    print(listdir(PYCACHEDIR))

[damianzd1]

Thank you so much for the fast reply. I installed it from source on my local machine and on a test EC2. Worked as needed. I tried to create an AWS lambda docker image, but when executing the lambda I got this error:

/var/lang/lib/python3.9/site-packages/stumpy/cache.py:67: UserWarning: Caching `numba` functions is purely for experimental purposes and should never be used or depended upon as it is not supported! All caching capabilities are not tested and may be removed/changed without prior notice. Please proceed with caution!
warnings.warn(CACHE_WARNING)
[ERROR] RuntimeError: cannot cache function '_compute_diagonal': no locator available for file '/var/lang/lib/python3.9/site-packages/stumpy/aamp.py'
Traceback (most recent call last):
  File "/var/lang/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/var/task/lambda_API_query_pattern.py", line 6, in <module>
    cache._enable()
  File "/var/lang/lib/python3.9/site-packages/stumpy/cache.py", line 72, in _enable
    func.enable_caching()
  File "/var/lang/lib/python3.9/site-packages/numba/core/dispatcher.py", line 863, in enable_caching
    self._cache = FunctionCache(self.py_func)
  File "/var/lang/lib/python3.9/site-packages/numba/core/caching.py", line 601, in __init__
    self._impl = self._impl_class(py_func)
  File "/var/lang/lib/python3.9/site-packages/numba/core/caching.py", line 337, in __init__
    raise RuntimeError("cannot cache function %r: no locator available "

I am trying to figure out what could've possibly gone wrong. I finished the docker file with a simple script (as you demonstrated) that did not error out during build time:

RUN python stumpy_compile.py

[seanlaw]

So, according to this numba issue, it looks like you are somehow unable to write (the cached functions) to the standard locations. Thus, to overcome this, you may need to set an environment variable called NUMBA_CACHE_DIR to point to a directory that you can write to, say. /tmp (see more details here).

Note that this is definitely venturing out beyond the support that we are able to provide.

[damianzd1]

Yes, I did try that as well, but sadly didn't work. I guess we will have to wait for Numbda to mature. You have been very helpful, thank you!
I am searching for a different topic, but I am unable to find existing discussions. Could you point me to one if I have missed it, or create a new one if needed? I am interested in stumpy.match but in multidimensional data. I want to match whole OHLC bars, not just a Close line. So I don't know if it is possible to provide a 4-dimensional query pattern and query a 4-dimensional time series? Even just 2 dimensions would be sufficient.

PS: ohhh lord, I've missed it. I think this is what I've been looking for: discussion

[seanlaw]

Yes, I did try that as well, but sadly didn't work. I guess we will have to wait for Numbda to mature. You have been very helpful, thank you!

Frankly, to be fair, I don't think this is a problem with Numba's maturity. The idea behind caching has been around since the beginning and so I think the problem has more to do with the limited (Docker) environment in which you are operating within that is preventing you from being able to write the cached files. This comment seemed to have worked but even without executing STUMPY, are you able to (within Docker):

1. Set the NUMBA_CACHE_DIR environment variable via ENV NUMBA_CACHE_DIR=/tmp/numba_cache
2. See if you can execute a command like echo "test" > $NUMBA_CACHE_DIR/test.txt
3. See if Python can see NUMBA_CACHE_DIR via import os; print(os.environ['NUMBA_CACHE_DIR'])

Otherwise, your environment is likely isn't setup correctly to operate with numba in a natural way. Unfortunately, I don't know much about Docker. IMHO, the fact that you are able to do all of this locally AND in EC2 actually speaks to something being weird/different in Docker and not likely an issue with numba (but I could be wrong).

Can you tell me where you are trying to write to on AWS lambda? I believe /tmp is the only allowable (fake) directory to write to

[seanlaw]

So I don't know if it is possible to provide a 4-dimensional query pattern and query a 4-dimensional time series? Even just 2 dimensions would be sufficient.

Note that a multi-dimensional matrix profile OR a multi-dimensional motif is not the same as four 1-dimensional matrix profiles/motifs stacked one on top of the other. There is a specific definition and so one needs to take special care here! See this tutorial for more details and please feel free to create a new discussion as the topic is changing.

[seanlaw]

If you compute a multi-dimensional matrix profile (using stumpy.mstump) then you'll want to use the equivalent multi-dimensional motif function stumpy.mmotifs here

[damianzd1]

Yes, you are correct. It's not an issue with Numba. I did some more testing. I found the issue but I have no idea what to do about it or even what exactly is happening. I managed to build a docker image from a base aws python lambda image. It works as intended when testing locally, but fails when uploaded to AWS and actually used by a lambda. The issue is that Aws runs the docker container with different user/group IDs. It doesn't matter if I set NUMBA_CACHE_DIR=/tmp. It runs it with User ID: 993, Group ID: 990 and it throws: no locator available error. Same if you run it locally with those ids. Locally if you run with default ids it functions as desired. I tried even copying the compiled functions to tmp, same result. One interesting thing that is happening is that for some reason even when stumpy is loading from the cache print("compiled funcs", len(cache._get_cache())) still prints out 0.

[seanlaw]

are you able to (within Docker):

1. Set the NUMBA_CACHE_DIR environment variable via ENV NUMBA_CACHE_DIR=/tmp/numba_cache
2. See if you can execute a command like echo "test" > $NUMBA_CACHE_DIR/test.txt
3. See if Python can see NUMBA_CACHE_DIR via import os; print(os.environ['NUMBA_CACHE_DIR'])

Were you able to at least confirm these things? If you can't even read/write to the desired directory from within the Docker container (i.e., outside of Python altogether) then all bets are off. Also, it sounds like AWS Lambda may have purge /tmp too without warning and so, even if you place cached files there, the caches may vanish without warning. Good luck!

[damianzd1]

Yes, all of them just not in the exact same format. But more than enough, I set ENV NUMBA_CACHE_DIR=/tmp (also tried different variations). I have no problem writing/reading files both during build time and after when the python session is running. with a normal docker run command the docker image performs flawlessly, All functions are cached (displayed them using numerous methods - tree /tmp, inside python, and so on). AWS docker image lets you test call it locally just like you would once it's uploaded to the cloud. Runs as desired, with no issues. The problem (as far as I am able to identify it) is that once uploaded to lambda, upon calling the lambda aws runs the container with different parameters for user & group IDs and then you get no locator available error. That is also re-creatable when testing locally by passing -u 993:990 param to the docker run command. Then you get the same error. I think this user doesn't have permission, even tho everything is stored and available in the /tmp directory. To be fair I have no idea what no locator available error means. I interpret it as cant read.

[seanlaw]

Okay. Let's take numba out of this. Are you able to do place a simple text file in /tmp and then have Python read that file from /tmp in AWS Lambda?

[damianzd1]

Okay... I am wrong again. I guess I trusted the awws docker local tests too much. You are correct. There is nothing in the /tmp once it runs on AWS servers and I wasn't able to read anything. I changed the target from /tmp to LAMBDA_TASK_ROOT which is /var/task and now I am able to read the test.txt inside the lambda running on aws. Tomorrow I will gather the strength to rebuild everything and try caching to /var/task. For reference this si the working setup:

ENV NUMBA_CACHE_DIR=${LAMBDA_TASK_ROOT}
RUN echo "test" > $NUMBA_CACHE_DIR/test.txt

[seanlaw]

Tomorrow I will gather the strength to rebuild everything and try caching to /var/task.

You can do it! You're so close. I can smell it! I believe in you @damianzd1 😄

[damianzd1]

Hmm, I didn't notice this before. The code errors out not when calling stumpy.stump, but when calling cache._enable(). All cache files are present and all conditions are met. The factor that makes the difference is -u 993:990 params when running the docker. Steps to reproduce:
Make a folder and place these files inside it:

Dockerfile

FROM public.ecr.aws/lambda/python:3.9
RUN /var/lang/bin/python3.9 -m pip install --upgrade pip
ENV NUMBA_CACHE_DIR=${LAMBDA_TASK_ROOT}
RUN echo "test" > $NUMBA_CACHE_DIR/test.txt

RUN yum -y install git
RUN git clone https://github.com/TDAmeritrade/stumpy.git

RUN cd stumpy && pip install -r requirements.txt
RUN cd stumpy && python -m pip install .

# Copy function code

ADD stumpy_compile.py ${LAMBDA_TASK_ROOT}/
RUN python stumpy_compile.py
RUN python stumpy_compile.py
COPY app.py ${LAMBDA_TASK_ROOT}/

CMD ["app.lambda_handler"]

stumpy_compile.py

# %%
import numpy as np
from stumpy import cache
import stumpy
import time
import os

# Get the effective user ID
user_id = os.geteuid()
print(f"User ID: {user_id}")

# Get the effective group ID
group_id = os.getegid()
print(f"Group ID: {group_id}")
# %%
cache._enable()
# %%
print("compiled funcs", len(cache._get_cache()))
print("NUMBA_CACHE_DIR", os.environ.get("NUMBA_CACHE_DIR", "NUMBA_CACHE_DIR missing"))
with open(os.environ.get("NUMBA_CACHE_DIR", "/tmp") + "/test.txt", "r") as file:
    contents = file.read()
    print("test.txt contents", contents)
# %%
T = np.random.rand(1000)
m = 50

start = time.time()
stumpy.stump(T, m)
elapsed_time = time.time() - start
print("Elapsed time: {:.2f} seconds".format(elapsed_time))

start = time.time()
stumpy.match(np.random.rand(50), np.random.rand(1000), max_distance=5.2, max_matches=20)
elapsed_time = time.time() - start
print("Elapsed time: {:.2f} seconds".format(elapsed_time))

print("compiled funcs", len(cache._get_cache()))

app.py

import os
import stumpy
from stumpy import cache
import numpy as np
import time


def list_files(startpath):
    for root, dirs, files in os.walk(startpath):
        level = root.replace(startpath, "").count(os.sep)
        indent = " " * 4 * (level)
        print("{}{}/".format(indent, os.path.basename(root)))
        subindent = " " * 4 * (level + 1)
        for f in files:
            print("{}{}".format(subindent, f))


print("NUMBA_CACHE_DIR", os.environ.get("NUMBA_CACHE_DIR", "NUMBA_CACHE_DIR missing"))
with open(os.environ.get("NUMBA_CACHE_DIR") + "/test.txt", "r") as file:
    contents = file.read()
    print("test.txt contents", contents)
list_files(os.environ.get("NUMBA_CACHE_DIR", "/var/task"))

print("enable cache")
cache._enable()
print("enable cache done")


def lambda_handler(event=None, context=None):
    T = np.random.rand(1000)
    m = 50

    start = time.time()
    stumpy.stump(T, m)
    elapsed_time = time.time() - start
    print("Elapsed time: {:.2f} seconds".format(elapsed_time))

    return {
        "statusCode": 200,
        "body": "Elapsed time: {:.2f} seconds".format(elapsed_time),
    }


if __name__ == "__main__":
    print(lambda_handler())

cd into your directory and build:

docker build -t stumpy_lambda .

Run locally, runs successfully (performs as desired):

docker run  -p 9000:8080 --rm --name lambda stumpy_lambda

Run locally, the way aws lambda service runs the container (results in error):

docker run -u 993:990 -p 9000:8080 --rm --name lambda stumpy_lambda

Call the lambda function locally:
Windows powershell:

Invoke-RestMethod -Method Post -Uri "http://localhost:9000/2015-03-31/functions/function/invocations" -Body '{}'

Linux curl:

curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

Running on aws results in the same error as running locally with -u 993:990:

raise RuntimeError("cannot cache function %r: no locator available "ne 337, in __init__e_caching/lang/lib/python3.9/site-packages/stumpy/aamp.py'

the docker log:

[ERROR] RuntimeError: cannot cache function '_compute_diagonal': no locator available for file '/var/lang/lib/python3.9/site-packages/stumpy  File "/var/lang/lib/python3.9/importlib/__init__.py", line 127, in import_modu  File "/var/lang/lib/python3.9/site-packages/stumpy/cache.py", line 72, in _ena  File "/var/lang/lib/python3.9/site-packages/numba/core/dispatcher.py", line 86  File "/var/lang/lib/python3.9/site-packages/numba/core/caching.py", line 601,   File "/var/lang/lib/python3.9/site-packages/numba/core/caching.py", line 337,     raise RuntimeError("cannot cache function %r: no locator available "

[seanlaw]

What about this comment:

What does this give you:
python -c "from numba.caching import _UserProvidedCacheLocator; print(_UserProvidedCacheLocator(lambda x:x, 'string').get_cache_path())"

[damianzd1]

I can't run that command even in my local python shell(tried multiple variations):

ModuleNotFoundError: No module named 'numba.caching'

Does it work for you?

[seanlaw]

ModuleNotFoundError: No module named 'numba.caching'

It looks like that comment is a tad outdated and caching has been moved into numba.core.caching

[damianzd1]

It returns /var/task/task_c018fdd87226113727f921403bd2e55efd69dd75 with or without -u 993:990 in local lambda docker container test and on aws service

[seanlaw]

Hmm, it feels like it should just print /var/task and I'm not sure why it's appending task_c018fdd87226113727f921403bd2e55efd69dd75 and if that is expected behavior. I think you should post this question on the numba issues tracker and reference this conversation.