Skip to content

Conversation

@gmogames
Copy link

@gmogames gmogames commented Oct 7, 2025

Summary

This PR fixes an issue that prevented multiple buildkite-builder run commands from being executed sequentially within the same container/process, enabling more efficient CI/CD workflows.

Problem

Previously, running multiple buildkite-builder run commands in sequence (e.g., via && chaining) would fail because the first run would set job metadata that couldn't be overwritten by subsequent runs, causing the error:
Error: Could not set job meta-data: The job meta-data key "buildkite_builder_uploaded_[pipeline_name]" already exists

Solution

  • Modified metadata handling to allow multiple pipeline uploads within the same container lifecycle
  • Each pipeline upload now properly manages its own metadata without conflicts
  • Maintains backward compatibility with single pipeline usage

Usage

Before (would fail)

# This would fail on the second command
buildkite-builder run pipeline.deploy && 
buildkite-builder run pipeline.unit.tests

After (now works)

# Multiple pipelines can now be uploaded sequentially
buildkite-builder run pipeline.deploy && \
buildkite-builder run pipeline.unit.tests && \
buildkite-builder run pipeline.integration

Benefits

  • Container Efficiency: Allows multiple pipelines to be uploaded from a single container
  • Performance: Eliminates the need to spin up separate containers for each pipeline
  • Flexibility: Enables complex CI/CD workflows with conditional pipeline uploads

Testing

✅ Tested sequential pipeline uploads within same container
✅ Verified backward compatibility with single pipeline usage
✅ Confirmed metadata handling doesn't cause conflicts
image

Impact

  • Reduces CI/CD execution time by avoiding multiple container startups
  • Enables more complex pipeline orchestration patterns
  • Fully backward compatible with existing single-pipeline workflows

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant