Blog/apache spark #74
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for infraspec ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
arihant-2310
reviewed
Feb 7, 2025
|
|
||
| ### 2\. Spark Executors | ||
|
|
||
| Executors (process)are the “workers” that actually do the processing. They take instructions from the driver, execute the tasks, and send back the results. Every Spark application gets its own set of executors, which run on different machines. They are responsible for completing the tasks, saving data, reporting results, and re-running any tasks that fail. |
There was a problem hiding this comment.
Fix the spacing between (process) and are.
arihant-2310
reviewed
Feb 7, 2025
content/blog/apache-spark-unleashing-big-data-with-rdds-dataframes-and-beyond.md
Outdated
Show resolved
Hide resolved
arihant-2310
reviewed
Feb 7, 2025
content/blog/apache-spark-unleashing-big-data-with-rdds-dataframes-and-beyond.md
Outdated
Show resolved
Hide resolved
arihant-2310
reviewed
Feb 7, 2025
content/blog/apache-spark-unleashing-big-data-with-rdds-dataframes-and-beyond.md
Outdated
Show resolved
Hide resolved
arihant-2310
reviewed
Feb 7, 2025
content/blog/apache-spark-unleashing-big-data-with-rdds-dataframes-and-beyond.md
Outdated
Show resolved
Hide resolved
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| That’s when I came across Apache Spark. It handles large datasets by distributing the work across multiple machines. | ||
|
|
||
| When I ran my first Spark job, I was honestly amazed. It processed my data way faster than Pandas. **The coolest part?** The driver-executor model. The driver assigns tasks, and the executors do the heavy lifting. If something goes wrong, Spark retries only the failed tasks instead of starting over, offering fault tolerance, and efficient distribution. Plus, it works well with cluster managers like **YARN** and **Kubernetes**, making it easy to scale up. |
There was a problem hiding this comment.
Grammar issue: driver-executor
Suggestion: driver–executor
|
|
||
| ### Cluster Managers and Storage Systems | ||
|
|
||
| Apache Spark can run on multiple cluster managers like, Standalone which is Spark’s built-in resource manager for small to medium-sized clusters, Hadoop YARN (Yet another resource navigator), Apache Mesos, Kubernetes for orchestrating Spark workloads in containerized environments. |
There was a problem hiding this comment.
Spelling issue: another
Suggestion: Another
|
|
||
| <h3 id="rdd"> RDDs (Resilient Distributed Datasets)</h3> | ||
|
|
||
| They are the fundamental building block of Spark's older API, introduced in the Spark 1.x series. While RDDs are still available in Spark 2.x and beyond, they are no longer the default API due to the introduction of higher-level abstractions like DataFrames and Datasets. However, every operation in Spark ultimately gets compiled down to RDDs, making it important to understand their basics. The Spark UI also displays job execution details in terms of RDDs, so having a working knowledge of them is essential for debugging and optimization. |
There was a problem hiding this comment.
Spelling issue: more
Suggestion: More
arihant-2310
approved these changes
Feb 7, 2025
|
@arihant-2310 , I have made the requested changes. |
Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.