BLD/RLS: Update wheels to include GDAL 3.9.2 and libkml #451
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -54,26 +54,34 @@ jobs: | |
| needs: [build-sdist] | ||
| runs-on: ubuntu-latest | ||
| container: | ||
| image: "ghcr.io/osgeo/gdal:ubuntu-small-3.9.2" | ||
|
|
||
| steps: | ||
| - name: Install packages | ||
| run: | | ||
| apt-get update && apt-get install -y build-essential python3-dev | ||
|
|
||
| - name: Create virtual environment | ||
| run: | | ||
| curl -LsSf https://astral.sh/uv/install.sh | sh | ||
| . $HOME/.cargo/env | ||
| uv venv .venv | ||
| echo "VIRTUAL_ENV=.venv" >> $GITHUB_ENV | ||
| echo "$PWD/.venv/bin" >> $GITHUB_PATH | ||
|
There was a problem hiding this comment. What is this doing exactly? (is it common practice documented somewhere, or can we add some comments?) From quickly checking the uv docs (https://docs.astral.sh/uv/pip/environments/), my understanding is that when you use the default There was a problem hiding this comment. But I suppose it is for the There was a problem hiding this comment. Correct, I believe I was running into issues with There was a problem hiding this comment. Just dropping a note that trying |
||
|
|
||
| - name: Download sdist from artifacts | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: pyogrio-sdist | ||
| path: wheelhouse | ||
|
|
||
| - name: Build from sdist and install test dependencies | ||
| shell: bash | ||
| run: | | ||
| uv pip install --no-cache wheelhouse/*.tar.gz | ||
| uv pip install --no-cache pytest pandas pyproj shapely>=2 | ||
| uv pip install --no-cache --no-deps geopandas | ||
| uv pip list | ||
|
|
||
| - name: Run tests | ||
| shell: bash | ||
|
|
@@ -83,7 +91,7 @@ jobs: | |
|
|
||
| build-wheels-linux: | ||
| name: Build wheels on Linux | ||
| runs-on: "ubuntu-latest" | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
|
|
@@ -103,7 +111,7 @@ jobs: | |
| uses: docker/build-push-action@v6 | ||
| with: | ||
| context: . | ||
| file: ci/manylinux_2_28_x86_64-vcpkg-gdal.Dockerfile | ||
| tags: manylinux-vcpkg-gdal:latest | ||
| push: false | ||
| load: true | ||
|
|
@@ -113,7 +121,7 @@ jobs: | |
| BUILDKIT_PROGRESS: plain | ||
|
|
||
| - name: Build wheels | ||
| uses: pypa/cibuildwheel@v2.20.0 | ||
|
|
||
| - uses: actions/upload-artifact@v4 | ||
| with: | ||
|
|
@@ -129,7 +137,7 @@ jobs: | |
| matrix: | ||
| include: | ||
| - os: "macos-12" | ||
| triplet: "x64-osx-dynamic-release" | ||
| arch: x86_64 | ||
| vcpkg_cache: "/Users/runner/.cache/vcpkg/archives" | ||
| vcpkg_logs: "/usr/local/share/vcpkg/buildtrees/**/*.log" | ||
|
|
@@ -160,7 +168,7 @@ jobs: | |
| path: | | ||
| ${{ matrix.vcpkg_cache }} | ||
| # bump the last digit to avoid using previous build cache | ||
| key: ${{ matrix.os }}-${{ matrix.arch }}-vcpkg-gdal3.9.2-cache0 | ||
|
|
||
| # MacOS build requires aclocal, which is part of automake, but appears | ||
| # to be missing in default image | ||
|
|
@@ -178,7 +186,7 @@ jobs: | |
| git reset --hard | ||
| # pull latest version to ensure the required commit with GDAL 3.8 is available | ||
| git pull | ||
| git checkout 73794ce5f63fd138fab999a22959ca7c6305d93c | ||
|
|
||
| - name: Install GDAL | ||
| env: | ||
|
There was a problem hiding this comment. There is a There was a problem hiding this comment. Don't we still need to use the custom triplets? |
||
|
|
@@ -196,7 +204,7 @@ jobs: | |
| path: ${{ matrix.vcpkg_logs }} | ||
|
|
||
| - name: Build wheels | ||
| uses: pypa/cibuildwheel@v2.20.0 | ||
| env: | ||
| # CIBW needs to know triplet for the correct install path | ||
| VCPKG_DEFAULT_TRIPLET: ${{ matrix.triplet }} | ||
|
|
@@ -215,17 +223,26 @@ jobs: | |
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: | ||
| [ | ||
| "ubuntu-latest", | ||
| "ubuntu-20.04", | ||
| "windows-latest", | ||
| "macos-12", | ||
| "macos-latest", | ||
| ] | ||
| python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] | ||
| include: | ||
| - os: "ubuntu-latest" | ||
| artifact: pyogrio-wheel-linux | ||
| - os: "ubuntu-20.04" | ||
| artifact: pyogrio-wheel-linux | ||
| - os: "windows-latest" | ||
| artifact: pyogrio-wheel-x64-windows-dynamic-release | ||
| - os: "macos-12" | ||
| artifact: pyogrio-wheel-x64-osx-dynamic-release | ||
| - os: "macos-latest" | ||
| artifact: pyogrio-wheel-arm64-osx-dynamic-release | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
|
|
@@ -236,8 +253,24 @@ jobs: | |
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| allow-prereleases: true | ||
|
|
||
| - name: Create virtual environment (Linux / MacOS) | ||
| if: ${{ runner.os != 'Windows' }} | ||
| shell: bash | ||
| run: | | ||
| curl -LsSf https://astral.sh/uv/install.sh | sh | ||
| . $HOME/.cargo/env | ||
| uv venv .venv | ||
| echo "VIRTUAL_ENV=.venv" >> $GITHUB_ENV | ||
| echo "$PWD/.venv/bin" >> $GITHUB_PATH | ||
|
|
||
| - name: Create virtual environment (Windows) | ||
| if: ${{ runner.os == 'Windows' }} | ||
| run: | | ||
| irm https://astral.sh/uv/install.ps1 | iex | ||
| uv venv .venv | ||
| "VIRTUAL_ENV=.venv" | Out-File -FilePath $env:GITHUB_ENV -Append | ||
| "$PWD/.venv/Scripts" | Out-File -FilePath $env:GITHUB_PATH -Append | ||
|
|
||
| - name: Download wheels from artifacts | ||
| uses: actions/download-artifact@v4 | ||
|
|
@@ -248,12 +281,12 @@ jobs: | |
| - name: Install dependencies and pyogrio wheel | ||
| shell: bash | ||
| run: | | ||
| uv pip install --no-cache -r ci/requirements-wheel-test.txt | ||
| if [ ${{ matrix.python-version }} != "3.12" ]; then | ||
| uv pip install --no-cache --no-deps geopandas | ||
| fi | ||
| uv pip install --no-cache --pre --no-index --find-links wheelhouse pyogrio | ||
| uv pip list | ||
|
|
||
| - name: Run tests | ||
| shell: bash | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
brendan-ward marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| FROM quay.io/pypa/manylinux_2_28_x86_64:2024-07-15-c746fd8 | ||
|
|
||
| # building openssl needs IPC-Cmd (https://github.com/microsoft/vcpkg/issues/24988) | ||
| RUN dnf -y install curl zip unzip tar ninja-build perl-IPC-Cmd | ||
|
|
||
| RUN git clone https://github.com/Microsoft/vcpkg.git /opt/vcpkg && \ | ||
| git -C /opt/vcpkg checkout 73794ce5f63fd138fab999a22959ca7c6305d93c | ||
|
|
||
| ENV VCPKG_INSTALLATION_ROOT="/opt/vcpkg" | ||
| ENV PATH="${PATH}:/opt/vcpkg" | ||
|
|
||
| ENV VCPKG_DEFAULT_TRIPLET="x64-linux-dynamic-release" | ||
|
|
||
| # mkdir & touch -> workaround for https://github.com/microsoft/vcpkg/issues/27786 | ||
| RUN bootstrap-vcpkg.sh && \ | ||
| mkdir -p /root/.vcpkg/ $HOME/.vcpkg && \ | ||
| touch /root/.vcpkg/vcpkg.path.txt $HOME/.vcpkg/vcpkg.path.txt && \ | ||
| vcpkg integrate install && \ | ||
| vcpkg integrate bash | ||
|
|
||
| COPY ci/custom-triplets/x64-linux-dynamic-release.cmake opt/vcpkg/custom-triplets/x64-linux-dynamic-release.cmake | ||
| COPY ci/vcpkg.json opt/vcpkg/ | ||
|
|
||
| ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/vcpkg/installed/x64-linux-dynamic-release/lib" | ||
| RUN vcpkg install --overlay-triplets=opt/vcpkg/custom-triplets \ | ||
| --feature-flags="versions,manifests" \ | ||
| --x-manifest-root=opt/vcpkg \ | ||
| --x-install-root=opt/vcpkg/installed && \ | ||
| vcpkg list | ||
|
|
||
| # setting git safe directory is required for properly building wheels when | ||
| # git >= 2.35.3 | ||
| RUN git config --global --add safe.directory "*" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you know why this is now needed? (was there a change in the docker image?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Originally we were installing
python3-pipwhich also pulled inbuild-essentialandpython3-devas dependencies. We now need to install them directly.