Added in-depth comments highlighting lakeFS use cases in spark-demo #212
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
omertzionitreeverse
added
the
documentation
|
Added a summary for airflow demos to improve clarity. |
iddoavn
reviewed
Jul 11, 2024
00_notebooks/spark-demo.ipynb
Outdated
| "id": "7a80e29b", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The S3A gateway makes it easy to use data tools like Spark with S3 storage. It handles large datasets efficiently, saves costs with affordable storage, and organizes data neatly for better management. Combined with lakeFS, one can safely experiment with data versions without risking the original. In the above we read the Parquet file from the source branch using the S3A Gateway. After reading, the data is stored in lakeFS, which allows us to version control and manage our data efficiently. This way, we can safely read and analyze the data without worrying about accidental changes.\n" |
There was a problem hiding this comment.
How about something like:
This example demonstrates using the lakeFS S3 gateway to interact with Spark.
The S3 gateway is an endpoint provided by lakeFS that implements a subset of the S3 API.
It acts as a bridge between Spark and the underlying object storage, providing a simplified way to access data.
Alternatively, the LakeFS Spark client can be used for direct data flow to/from object storage, potentially offering improved performance and security.
For more information on the LakeFS Spark client, refer to: https://docs.lakefs.io/reference/spark-client.html
kesarwam
approved these changes
Jul 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a summary as an introduction to assist with clarity. Also added descriptions for sections using the S3A gateway and a lakeFS 0-copy branching explanation. Purpose is to enhance clarity when going through demos.