-
Notifications
You must be signed in to change notification settings - Fork 1
Move creation of KV store dictionary to python layer #93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
5a6e652
1720462
5d8d1cd
96b98f8
0cd66da
da5f0de
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,29 +1,27 @@ | ||
name: Run Tests | ||
|
||
on: | ||
push: | ||
branches: | ||
- main | ||
- branch-bug | ||
- update-test-branch | ||
- test-branch7 | ||
- test-branch8 | ||
#on: | ||
# push: | ||
# branches: | ||
#- main | ||
#- test-branch-pyDict | ||
|
||
jobs: | ||
test: | ||
runs-on: ${{matrix.os}} | ||
strategy: | ||
matrix: | ||
os: [ubuntu-latest, windows-latest, macos-14] | ||
matlab-version: ['R2023b','R2024a'] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If 23b is working fine after your fix, please make sure to update this section: https://github.com/mathworks/MATLAB-support-for-Zarr-files?tab=readme-ov-file#mathworks-products-httpswwwmathworkscom There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point! Will check once we are able to run the tests on multiple releases, and update then (if no issues surface in the tests) |
||
|
||
steps: | ||
- name: Checkout code | ||
uses: actions/checkout@v2 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you please update this to "v4" instead of "v2"? |
||
|
||
- name: Set up MATLAB R2024a | ||
- name: Set up MATLAB | ||
uses: matlab-actions/setup-matlab@v2 | ||
with: | ||
release: R2024a | ||
release: ${{matrix.matlab-version}} | ||
|
||
- name: Set up Python | ||
uses: actions/setup-python@v5 | ||
|
@@ -36,7 +34,7 @@ jobs: | |
pip install numpy==1.26.4 tensorstore==0.1.73 | ||
|
||
- name: Run tests | ||
uses: matlab-actions/run-tests@v2 | ||
uses: matlab-actions/run-tests@v4 | ||
with: | ||
select-by-folder: 'test' | ||
code-coverage-cobertura: cobertura.xml | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,6 +7,32 @@ | |
import numpy as np | ||
import tensorstore as ts | ||
|
||
def createKVStore(isRemote, objPath, bucketName="") -> dict: | ||
""" | ||
Creates a KV store (a python dictionary) for reading or writing | ||
a Zarr file | ||
|
||
Parameters: | ||
- isRemote (bool): whether the resource to be accessed with this | ||
KV store is remote (S3) or local | ||
- objPath (str): path to local Zarr file or to S3 object | ||
- bucketName (str): If file is remote, this should be the S3 bucket | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. consistency for using "S3" or "s3"? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point - updating to "S3" for consistency |
||
name | ||
|
||
Returns: | ||
- KVStore (dict): Key-Value store as required by tensorstore to work | ||
with Zarr | ||
""" | ||
KVStore = dict(path=objPath); | ||
|
||
if isRemote: | ||
KVStore['driver'] = 's3' | ||
KVStore['bucket'] = bucketName | ||
else: | ||
KVStore['driver'] = 'file' | ||
|
||
return KVStore | ||
|
||
def createZarr(kvstore_schema, data_shape, chunk_shape, tstoreDataType, zarrDataType, compressor, fillvalue): | ||
""" | ||
Creates a new Zarr array and writes data to it. | ||
|
@@ -72,4 +98,4 @@ def readZarr (kvstore_schema): | |
|
||
# Read a subset of the data | ||
data = zarr_file[...].read().result() | ||
return data | ||
return data |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -198,14 +198,12 @@ function makeZarrGroups(existingParentPath, newGroupsPath) | |
% Extract the S3 bucket name and path | ||
[bucketName, objectPath] = obj.extractS3BucketNameAndPath(obj.Path); | ||
% Create a Python dictionary for the KV store driver | ||
RemoteStoreSchema = dictionary(["driver", "bucket", "path"], ["s3", bucketName, objectPath]); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Less is more. Nice. |
||
obj.KVStoreSchema = py.dict(RemoteStoreSchema); | ||
obj.KVStoreSchema = py.ZarrPy.createKVStore(obj.isRemote, objectPath, bucketName); | ||
|
||
else % Local file | ||
% use full path | ||
obj.Path = Zarr.getFullPath(path); | ||
FileStoreSchema = dictionary(["driver", "path"], ["file", obj.Path]); | ||
obj.KVStoreSchema = py.dict(FileStoreSchema); | ||
obj.KVStoreSchema = py.ZarrPy.createKVStore(obj.isRemote, obj.Path); | ||
end | ||
end | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving as a temporary change due to the ongoing outage, but the tests were qualified locally.