Stuck on the start line

[waterworthd-cim]

I'm interested in marquez/open-lineage but I cannot get it to work. I'm working through the "getting started" (https://openlineage.io/getting-started) but haven't made it very far. Any help appreciated

I'm running on macos (m3) using colima. The first issue was ./docker/up.sh failed until I ran ./docker/up.sh --no-web to create the database. I don't think this is expected but looked to me like there was some sort of race - but I didn't copy the error message.

Now ./docker/up.sh seems to work but every time I try and create a job I get a 403?

curl -X POST http://localhost:5000/api/v1/lineage \
  -i -H 'Content-Type: application/json' \
  -d '{
        "eventType": "START",
        "eventTime": "2020-12-28T19:52:00.001+10:00",
        "run": {
          "runId": "0176a8c2-fe01-7439-87e6-56a1a1b4029f"
        },
        "job": {
          "namespace": "my-namespace",
          "name": "my-job"
        },
        "inputs": [{
          "namespace": "my-namespace",
          "name": "my-input"
        }],
        "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
        "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
      }'
HTTP/1.1 403 Forbidden
Content-Length: 0
Server: AirTunes/860.7.1
X-Apple-ProcessingTime: 0
X-Apple-RequestReceivedTimestamp: 17958135

I also tried ./docker/up.sh --seed and now there's a never ending loop of stack traces in the logs

marquez-api                 | ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
marquez-api                 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
marquez-api                 | ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
marquez-api                 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
marquez-api                 | ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
marquez-api                 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
marquez-api                 | ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
marquez-api                 | ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
marquez-api                 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
marquez-api                 | ! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322)
marquez-api                 | ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
marquez-api                 | ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722)
marquez-api                 | ! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
marquez-api                 | ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
marquez-api                 | ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
marquez-api                 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
marquez-api                 | ! at org.eclipse.jetty.server.Server.handle(Server.java:516)
marquez-api                 | ! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
marquez-api                 | ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
marquez-api                 | ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
marquez-api                 | ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
marquez-api                 | ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
marquez-api                 | ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
marquez-api                 | ! at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
marquez-api                 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
marquez-api                 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
marquez-api                 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
marquez-api                 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
marquez-api                 | ! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
marquez-api                 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
marquez-api                 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
marquez-api                 | ! at java.base/java.lang.Thread.run(Thread.java:840)
marquez-api                 | INFO  [2025-07-21 05:59:48,066] marquez.logging.LoggingMdcFilter: status: 201
marquez-api                 | 172.19.0.5 - - [21/Jul/2025:05:59:48 +0000] "POST /api/v1/lineage HTTP/1.1" 201 0 "-" "Apache-HttpClient/4.5.14 (Java/17.0.14)" 224

[waterworthd-cim]

Also my log contains

marquez-search  | ./opensearch-docker-entrypoint.sh: line 70:   103 Killed                  "$@" "${opensearch_opts[@]}"
marquez-search exited with code 137

[Storm-Dev6]

Hi @waterworthd-cim 👋

Thanks for reporting your experience — sounds like you’ve run into a few startup issues. Let’s walk through some suggestions to help you get OpenLineage running smoothly with Docker + Colima on macOS M3.

✅ 1. ./docker/up.sh only working with --no-web
You're right — this behavior isn't expected, and it might be due to resource constraints or timing issues (race conditions) during container startup. On some machines, the web UI or marquez-api container may try to connect to the database before it's ready.

Suggestions:

Try increasing Docker's allocated memory/CPU via Colima:
colima start --memory 4 --cpu 3

Alternatively, run in two steps:

./docker/up.sh --no-web # Start backend & DB first
sleep 10
./docker/up.sh --web # Start web UI afterward

❌ 2. 403 Forbidden when posting lineage events
If you're getting 403 Forbidden when doing a POST to /api/v1/lineage, it's usually because:

The namespace doesn't exist yet

Marquez uses RBAC/auth controls — and in some configurations, default policies may block API access

Fix:

Try first registering the namespace and job using the proper API calls, or run the seeding script:

./docker/up.sh --seed

If you're still getting 403 after seeding, check for:

Token/auth headers required (check config or any reverse proxy)

Logs in marquez-api — sometimes stack traces hint at missing permissions or bad payloads

🔁 3. Endless stack traces & OpenSearch crash
This line from your logs shows OpenSearch was killed:

marquez-search | ./opensearch-docker-entrypoint.sh: line 70: 103 Killed ...
marquez-search exited with code 137

Exit code 137 usually means “Out Of Memory” (OOM) — OpenSearch needs a fair amount of RAM.

Suggestions:

Again, increase Colima memory:
colima start --memory 4 --cpu 3

Or disable the search container temporarily by editing docker-compose.yml and commenting out marquez-search

🧪 Summary Checklist
Increase Docker/Colima memory (OpenSearch needs it)

Use --no-web then --web as a workaround for DB timing

Make sure namespace/job exist before sending lineage events

Check marquez-api logs for permission or validation errors

Temporarily disable OpenSearch if not essential for testing

Let me know if you'd like a custom minimal docker-compose.override.yml for testing without search & web.

Hope this helps you get past the starting line! 🚀