build: replace rockylinux with chainguard/wolfi as a base image #423

christinestraub · 2024-06-11T20:32:10Z

Summary

Updates the Dockerfile to use the Chainguard wolfi-base image to reduce CVEs. Also adds a step in the docker publish job that scans the images and checks for CVEs before publishing.

Testing

Run make docker-build and make docker-start-api, then try:

from unstructured.partition.api import partition_via_api

elements = partition_via_api(
    filename=filename,
    api_url="http://localhost:8000/general/v0/general",
    api_key="<API-KEY>",
    strategy="hi_res",
)

print("\n\n".join([str(el) for el in elements]))

MthwRobinson

Looks good! Main comment is I don't think we need the download packages step now since all of that gets installed in the base images. We can remove that from unstructured now as well.

MthwRobinson · 2024-06-17T18:17:22Z

.github/workflows/ci.yml

@@ -110,5 +110,12 @@ jobs:
    - name: Test Dockerfile
      run: |
        source .venv/bin/activate
+        make docker-dl-packages


If we pull from base-images I don't think we'll need this anymore.

MthwRobinson · 2024-06-17T18:17:39Z

.github/workflows/docker-publish.yml

@@ -68,6 +68,7 @@ jobs:
        # Clear some space (https://github.com/actions/runner-images/issues/2840)
        sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/share/boost

+        make docker-dl-packages


Same here on download packages.

MthwRobinson · 2024-06-17T18:17:53Z

.gitignore

@@ -145,3 +145,6 @@ celerybeat-schedule.db

 # temporarily generated files by project-specific Makefile
 tmp*
+
+# APK packages for the docker build
+docker-packages/*


Same here on download packages.

MthwRobinson · 2024-06-17T18:18:04Z

CHANGELOG.md

@@ -1,3 +1,7 @@
+## 0.0.71-dev0
+
+* replace rockylinux with chainguard/wolfie as a base image for `amd64`


Small nit: wolfi instead of wolfie.

MthwRobinson · 2024-06-17T18:18:26Z

Makefile

@@ -64,6 +64,10 @@ DOCKER_IMAGE ?= pipeline-family-${PIPELINE_FAMILY}-dev:latest
 docker-build:
 	PIP_VERSION=${PIP_VERSION} PIPELINE_FAMILY=${PIPELINE_FAMILY} PIPELINE_PACKAGE=${PIPELINE_PACKAGE} ./scripts/docker-build.sh

+.PHONY: docker-dl-packages


Same here on download packages

MthwRobinson · 2024-06-17T18:18:40Z

scripts/docker-dl-packages.sh

@@ -0,0 +1,22 @@
+#!/bin/bash


Same here on download packages.

MthwRobinson

LGTM!

…indows (#2) ## PR Summary 1. Merged changes from upstream. 2. Update `unstructuredio_api.spec`. 3. Update `unstructuredio_api.py`. 4. Add additional setup dependencies to the `docs/Windows.md`. * build(deps): version bumps for maintenance (Unstructured-IO#424) ### Summary Version bumps for regular maintenance and to address moderate CVEs from security scans. - bump `unstructured` to `0.14.6` - bump `unstructured-inference` to `0.7.35` * build: replace rockylinux with chainguard/wolfi as a base image (Unstructured-IO#423) ### Summary Updates the Dockerfile to use the Chainguard wolfi-base image to reduce CVEs. Also adds a step in the docker publish job that scans the images and checks for CVEs before publishing. ### Testing Run `make docker-build` and `make docker-start-api`, then try: ``` from unstructured.partition.api import partition_via_api elements = partition_via_api( filename=filename, api_url="http://localhost:8000/general/v0/general", api_key="<API-KEY>", strategy="hi_res", ) print("\n\n".join([str(el) for el in elements])) ``` * fix: build and push workflow failing due to missing `-f` option `buildx build` command (Unstructured-IO#425) I noticed that images on main branch are failing to build (and push) due to missing `-f` parameter in `docker buildx build`. By default it expects `Dockerfile` to exist, but we only have `Dockerfile-amd64` and `Dockerfile-arm64` ![image](https://github.com/Unstructured-IO/unstructured-api/assets/64484917/4527165a-909e-498d-b0ee-8bba4b1a13e4) --------- Co-authored-by: christinestraub <christinemstraub@gmail.com> * fix: update SHA for the base images (both architectures) after `base-images` repo update (Unstructured-IO#427) build and publish CI steps are failing, because the base images have changed in quay (their SHAs) ![image](https://github.com/Unstructured-IO/unstructured-api/assets/64484917/fc4e9aac-0820-4c90-9ad9-68cc6d9aad03) ![image](https://github.com/Unstructured-IO/unstructured-api/assets/64484917/fafe2ca4-dab2-4610-a26b-a7a4d56723a5) * fix: revert to rockylinux SHA that works (arm64) (Unstructured-IO#428) unnecessary SHA update introduced in Unstructured-IO#427 that needs to be reverted * fix: re-add `DOCKER_IMAGE` env var in `Test image` step (Unstructured-IO#429) shell syntax error occurs in docker-publish.yml workflow * fix: invalid env var setting in `docker-publish` workflow (Unstructured-IO#430) bug introduced in previous PR causing build failure on main * fix: `docker-publish` workflow failing on main due to inexisting `ARCH` env var (Unstructured-IO#431) * build(deps): bump dependency versions (Unstructured-IO#434) ### Summary Bumps dependency versions for the API. Closes Unstructured-IO#432. * fix/Fix MS Office filetype errors and harden docker smoketest (Unstructured-IO#436) # Changes **Fix for docx and other office files returning `{"detail":"File type None is not supported."}`** After moving to the wolfi base image, the `mimetypes` lib no longer knows about these file extensions. To avoid issues like this, let's add an explicit mapping for all the file extensions we care about. I added a `filetypes.py` and moved `get_validated_mimetype` over. When this file is imported, we'll call `mimetypes.add_type` for all file extensions we support. **Update smoke test coverage** This bug snuck past because we were already providing the mimetype in the docker smoke test. I updated `test_happy_path` to test against the container with and without passing `content_type`. I added some missing filetypes, and sorted the test params by extension so we can see when new types are missing. # Testing The new smoke test will verify that all filetypes are working. You can also `make docker-build && make docker-start-api`, and test out the docx in the sample docs dir. On `main`, this file will give you the error above. ``` curl 'http://localhost:8000/general/v0/general' \ --form 'files=@"fake.docx"' ``` * merge main; validated format: xml, txv, csv, xml, json, html, docs, docx, ppt, pptx, xlsx, xls, pdf * compilable setting * update Windows markdown * Disable debug mode in `unstructuredio_api.spec` * Enable pdf test case in `test_app.py` --------- Co-authored-by: Christine Straub <christinemstraub@gmail.com> Co-authored-by: Michał Martyniak <64484917+micmarty-deepsense@users.noreply.github.com> Co-authored-by: Matt Robinson <mrobinson@unstructured.io> Co-authored-by: Austin Walker <austin@unstructured.io> Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>

christinestraub added 11 commits June 10, 2024 14:42

refactor: rename Dockerfile to Dockerfile-arm64

4a2f387

refactor: add Dockerfile-amd64

8abd178

add script to download apk packages

15bda57

switch Dockerfile-amd64 over to wolfi

01aa66a

download packages in ci

2de86bd

add ci job to scan the image for vulnerabilities

1475162

update docker-build.sh

22fbdab

back to libreoffice7

ce7f6ca

pull from wolfi base image

2e1bb39

pull from wolfi base image

7dec484

use ENV PIPELINE_FAMILY when scanning image

e6fbbf6

christinestraub marked this pull request as ready for review June 14, 2024 22:12

christinestraub added 2 commits June 14, 2024 15:17

Merge branch 'refs/heads/main' into OSS-12/wolfi-base-image

45bde1b

chore: update changelog & version

4badcc4

christinestraub requested review from MthwRobinson and micmarty-deepsense June 14, 2024 22:33

MthwRobinson suggested changes Jun 17, 2024

View reviewed changes

christinestraub changed the title ~~build: replace rockylinux with chainguard/wolfie as a base image~~ build: replace rockylinux with chainguard/wolfi as a base image Jun 17, 2024

remove download packages step

fdef839

MthwRobinson approved these changes Jun 17, 2024

View reviewed changes

christinestraub merged commit 2bdd52a into main Jun 17, 2024
6 checks passed

christinestraub deleted the OSS-12/wolfi-base-image branch June 17, 2024 20:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

build: replace rockylinux with chainguard/wolfi as a base image #423

build: replace rockylinux with chainguard/wolfi as a base image #423

Uh oh!

christinestraub commented Jun 11, 2024 •

edited

Loading

Uh oh!

MthwRobinson left a comment

Uh oh!

MthwRobinson Jun 17, 2024

Uh oh!

MthwRobinson Jun 17, 2024

Uh oh!

MthwRobinson Jun 17, 2024

Uh oh!

MthwRobinson Jun 17, 2024

Uh oh!

MthwRobinson Jun 17, 2024

Uh oh!

MthwRobinson Jun 17, 2024

Uh oh!

MthwRobinson left a comment

Uh oh!

Uh oh!

Uh oh!

		@@ -1,3 +1,7 @@
		## 0.0.71-dev0

		* replace rockylinux with chainguard/wolfie as a base image for `amd64`

build: replace rockylinux with chainguard/wolfi as a base image #423

build: replace rockylinux with chainguard/wolfi as a base image #423

Uh oh!

Conversation

christinestraub commented Jun 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

MthwRobinson left a comment

Choose a reason for hiding this comment

Uh oh!

MthwRobinson Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

MthwRobinson Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

MthwRobinson Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

MthwRobinson Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

MthwRobinson Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

MthwRobinson Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

MthwRobinson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

christinestraub commented Jun 11, 2024 •

edited

Loading