Skip to content

Commit f3bad32

Browse files
rollroll
authored
feat!: Improved packages structure, public API (#211)
* Rebased on p-all * Updated the link in readmes * Updated polars imports * Fixed tests * Simplify DataType usage * Added Frame type * Rebased on using frame * Hide node warnings in vitest * Moved session to cli root * Rebased on @minify for profiles * Renamed dp binary to dpkit * Updated pr template * Added load folder spec * Added save folder spec * Fixed tests * Added DataGrid spec * Added TableGrid spec * Fixed CI tests * Reworked error/exception/report in core * Updated file * Renamed node to system * Updated table * Updated lib * Rebased table on inspect functions * Updated validateFile * Moved fk checks to lib * Renamed system to platform in core * Renamed dpkit to system * Simplified MetadataError * Bootstrapped document package * Added DocumentError * Updated cell/jsonSchema error * Added resource.jsonSchema * Implemented validateDocument * Fixed validateDocument * Fixed types in tests * Enabled type checking in specs * Renamed result to report * Fixed types * Added BoundError and UnboundError * Bootstrapped jsonSchema module * Split jsonSchema/profile * Renamed jsonSchema to json * FIxed table * Fixed types * Added inspectJsonSchema test * Removed DpkitException * FIxed assert specs * Updated document/table * Fixed tests * Rebased on vitest-polly * Upgraded vitest-polly * Merged table/xlsx * Merged table/parquet * Merged table/ods * Merged table/json * Merged table/inline * Merged table/arrow * Merged table/csv * Updated inline structure * Updated json structure * Updated ods structure * Updated parquet structure * Updated xlsx structure * Renamed @dpkit/core -> @dpkit/metadata * Renamed @dpkit/file -> @dpkit/dataset * Rebased on DatasetPlugin * Merged dataset/datahub * Merged dataset/ckan * Merged dataset/folder * Merged dataset/github * Merged dataset/zenodo * Fixed metadata rootDir * Merged dataset/zip * Renamed cli -> terminal * Bootstrapped video package * Bootstrapped audio package * Bootstrapped image package * Renamed lib -> library * Improved plugin system * Revert "Improved plugin system" This reverts commit 66fbff7. * Removed metadata plugin * Moved html to document * Merged document/markdown * Switch main branch to pre-release * Added package.json:files * Fixed dependency versions * Fixed CI * Fixed terminal compilation * Fixed compile * Implemented DescriptorPlugin * Renamed to inferName/Format * Removed polars dep from library * Fixed compilation errors * Fixed dpkit * Bump nodejs-polars@0.22.2 * Explicit metadata exports * Moved mergePackages to dataset * Fixed errors * Updated doc references to new public API * Added library:Plugin * Explicit dataset exports * Renamed infer file functions * Updated infer file signatures * Updated infer file api * Explicit table exports * Explicit document exports * Updated deps * Explicit library exports * Added more library re-exports * Added table explore spec * Added script specs * Added other script specs * Removed useRecording * Added resoruce script spec * Added schema script spec * Added schema explore spec * Added resource explore spec * Added dialect explore spec * Added package explore spec * Fixed coverage ignore pattern
1 parent d44510b commit f3bad32

File tree

807 files changed

+8276
-5543
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

807 files changed

+8276
-5543
lines changed

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
- fixes #
1+
- Fixes #
22

33
---
44

‎.github/workflows/general.yaml‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ jobs:
3434
uses: actions/upload-artifact@v4
3535
with:
3636
name: binaries
37-
path: cli/compile/binaries/*.zip
37+
path: terminal/compile/binaries/*.zip
3838

3939
test-linux:
4040
needs: [test-compile]
@@ -70,9 +70,10 @@ jobs:
7070
uses: actions/download-artifact@v5
7171
with:
7272
name: binaries
73-
path: cli/compile/binaries
73+
path: terminal/compile/binaries
7474
- name: Test Binaries
75-
run: unzip -j cli/compile/binaries/*linux-x64.zip && ./dp --version
75+
working-directory: terminal/compile/binaries
76+
run: unzip -j *linux-x64.zip && ./dpkit --version
7677

7778
services:
7879
postgres:
@@ -119,9 +120,10 @@ jobs:
119120
uses: actions/download-artifact@v5
120121
with:
121122
name: binaries
122-
path: cli/compile/binaries
123+
path: terminal/compile/binaries
123124
- name: Test Binaries
124-
run: unzip -j cli/compile/binaries/*macos-arm64.zip && ./dp --version
125+
working-directory: terminal/compile/binaries
126+
run: unzip -j *macos-arm64.zip && ./dpkit --version
125127

126128
test-windows:
127129
needs: [test-compile]
@@ -149,10 +151,11 @@ jobs:
149151
uses: actions/download-artifact@v5
150152
with:
151153
name: binaries
152-
path: cli/compile/binaries
154+
path: terminal/compile/binaries
153155
- name: Test Binaries
154156
shell: bash
155-
run: unzip -j cli/compile/binaries/*windows-x64.zip && ./dp.exe --version
157+
working-directory: terminal/compile/binaries
158+
run: unzip -j *windows-x64.zip && ./dpkit.exe --version
156159

157160
# We have to split the release step because for some reason
158161
# using semantic-release before compilation inflates the binary sizes
@@ -228,7 +231,7 @@ jobs:
228231
uses: softprops/action-gh-release@v2
229232
with:
230233
draft: false
231-
files: cli/compile/binaries/*.zip
234+
files: terminal/compile/binaries/*.zip
232235
tag_name: v${{ needs.release-draft.outputs.version }}
233236
env:
234237
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.gitignore‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,4 +69,3 @@ dist/
6969
.serena/
7070
.mcp.json
7171
.env
72-
dp

‎.releaserc.json‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
{
2-
"branches": ["main"],
2+
"branches": [
3+
{
4+
"name": "main",
5+
"prerelease": "rc"
6+
}
7+
],
38
"plugins": [
49
["@semantic-release/commit-analyzer", { "preset": "conventionalcommits" }],
510
[

‎CONTRIBUTING.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,14 @@ Thank you for your interest in contributing to dpkit! This document provides gui
99

1010
Project is a monorepo with the following packages:
1111

12-
- `@dpkit/core`: Core metadata functionality
13-
- `@dpkit/file`: File-related functionality
12+
- `@dpkit/metadata`: Core metadata functionality
13+
- `@dpkit/dataset`: File-related functionality
1414
- `@dpkit/table`: Table-related functionality
15-
- `@dpkit/cli`: Command-line interface for running tasks
1615
- `@dpkit/<name>`: Domain-specific functionality
17-
- `@dpkit/lib`: All-in-one package that re-exports all functionality
18-
- `dpkit`: Meta-package that re-exports `lib` and `cli`
16+
- `@dpkit/library`: All-in-one package that re-exports all functionality
17+
- `@dpkit/terminal`: Terminal interface for running tasks
18+
- `@dpkit/browser`: Browser-related functionality
19+
- `dpkit`: Meta-package that re-exports the underlying functionality
1920

2021
## Development Environment
2122

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
[![Coverage](https://img.shields.io/codecov/c/github/datisthq/dpkit/main)](https://codecov.io/gh/datisthq/dpkit)
55
[![Version](https://img.shields.io/npm/v/dpkit)](https://www.npmjs.com/package/dpkit)
66

7-
dpkit is a fast data management framework built on top of the Data Package standard and Polars DataFrames. It supports various formats like CSV, JSON, and Parquet and integrates with data platforms such as CKAN, Zenodo, and GitHub. For more information, please visit the [documentation portal](https://dpkit.dev).
7+
dpkit is a fast data management framework built on top of the Data Package standard and Polars DataFrames. It supports various formats like CSV, JSON, and Parquet and integrates with data platforms such as CKAN, Zenodo, and GitHub. For more information, please visit the [project's website](https://dpkit.app).
88

99
## Funding
1010

‎arrow/package.json‎

Lines changed: 0 additions & 36 deletions
This file was deleted.

‎lib/README.md‎ renamed to ‎audio/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
# @dpkit/lib
1+
# @dpkit/audio
22

3-
dpkit is a fast data management framework built on top of the Data Package standard and Polars DataFrames. It supports various formats like CSV, JSON, and Parquet and integrates with data platforms such as CKAN, Zenodo, and GitHub. For more information, please visit the [documentation portal](https://dpkit.dev).
3+
dpkit is a fast data management framework built on top of the Data Package standard and Polars DataFrames. It supports various formats like CSV, JSON, and Parquet and integrates with data platforms such as CKAN, Zenodo, and GitHub. For more information, please visit the [project's website](https://dpkit.app).

‎audio/index.ts‎

Whitespace-only changes.

‎folder/package.json‎ renamed to ‎audio/package.json‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
{
2-
"name": "@dpkit/folder",
2+
"name": "@dpkit/audio",
33
"type": "module",
44
"version": "0.0.0-dev",
55
"exports": "./build/index.js",
66
"sideEffects": false,
7+
"files": ["build"],
78
"license": "MIT",
89
"author": "Evgeny Karev",
910
"repository": "https://github.com/datisthq/dpkit",
@@ -18,14 +19,13 @@
1819
"validation",
1920
"quality",
2021
"fair",
21-
"folder"
22+
"audio"
2223
],
2324
"scripts": {
2425
"build": "tsc"
2526
},
2627
"dependencies": {
27-
"@dpkit/core": "workspace:*",
28-
"@dpkit/file": "workspace:*",
29-
"exit-hook": "^4.0.0"
28+
"@dpkit/metadata": "workspace:*",
29+
"@dpkit/dataset": "workspace:*"
3030
}
3131
}

0 commit comments

Comments
 (0)