Skip to content

add a callout for new glue executor #1751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 7, 2025
Merged

add a callout for new glue executor #1751

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e15727d
add a callout for new glue executor
HarshCasper May 2, 2025
6399c3b
fix
HarshCasper May 2, 2025
5723fa3
couple of fixes
HarshCasper May 5, 2025
1d054d1
grammar fixes
quetzalliwrites May 7, 2025
2995917
consistency in bullet points
quetzalliwrites May 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions content/en/user-guide/aws/glue/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,31 @@ linkTitle: Glue
description: Get started with Glue on LocalStack
tags: ["Pro image"]
---

## Introduction

The Glue API in LocalStack Pro allows you to run ETL (Extract-Transform-Load) jobs locally, maintaining table metadata in the local Glue data catalog, and using the Spark ecosystem (PySpark/Scala) to run data processing workflows.

LocalStack allows you to use the Glue APIs in your local environment.
The supported APIs are available on our [API coverage page](/references/coverage/coverage_glue/), which provides information on the extent of Glue's integration with LocalStack.

{{< callout >}}
LocalStack now includes a container-based Glue Job executor, allowing Glue jobs to run inside a Docker environment.
Previously, LocalStack used a pre-packaged binary with Spark and required components.
The new executor uses the `aws-glue-libs` Docker image, offering better production parity, faster startup, and more reliable execution.

It adds support for:

- Running Glue jobs in Docker containers
- Isolated execution environments for each job
- Executing multiple jobs in parallel
- Correct versioning of Spark, Hadoop, Python, Java, and related libraries
- Improved startup time & offline execution

To use it, set `GLUE_JOB_EXECUTOR=docker` and `GLUE_JOB_EXECUTOR_PROVIDER=v2` in your LocalStack configuration.
The new executor additionally deprecates older versions of Glue (`0.9`, `1.0`, `2.0`).
{{< /callout >}}

## Getting started

This guide is designed for users new to Glue and assumes basic knowledge of the AWS CLI and our [`awslocal`](https://github.com/localstack/awscli-local) wrapper script.
Expand Down