Apache Spark 4.0.0

Author: jaceklaskowski

docs/index.md

@@ -10,12 +10,13 @@ hide:
 
 Welcome to **The Internals of {{ book.title }}** online book! 🤙
 
-I'm [Jacek Laskowski](https://pl.linkedin.com/in/jaceklaskowski), a Freelance Data(bricks) Engineer specializing in
+I'm [Jacek Laskowski](https://pl.linkedin.com/in/jaceklaskowski), a Freelance Data(bricks) Engineer 🧱 specializing in
 [Apache Spark](https://books.japila.pl/apache-spark-internals/) (incl. [Spark SQL](https://books.japila.pl/spark-sql-internals/) and [Spark Structured Streaming](https://books.japila.pl/spark-structured-streaming-internals/)),
 [Delta Lake](https://books.japila.pl/delta-lake-internals/),
 [Unity Catalog](https://books.japila.pl/unity-catalog-internals/),
-[Databricks](https://www.databricks.com/),
-and [Apache Kafka](https://books.japila.pl/kafka-internals/) (incl. [Kafka Streams](https://books.japila.pl/kafka-streams-internals/)) with brief forays into a wider data engineering space (mostly during [Warsaw Data Engineering](https://www.meetup.com/Warsaw-Data-Engineering/) meetups).
+[MLflow](https://mlflow.org/),
+[Databricks](https://www.databricks.com/)
+with brief forays into a wider data engineering space (mostly during [Warsaw Data Engineering](https://www.meetup.com/Warsaw-Data-Engineering/) meetups).
 
 I'm very excited to have you here and hope you will enjoy exploring the internals of {{ book.title }} as much as I have.
 

mkdocs.yml

@@ -106,22 +106,27 @@ extra:
     spark_core: https://books.japila.pl/apache-spark-internals
     structured_streaming: https://books.japila.pl/spark-structured-streaming-internals
   hadoop:
-    version: 3.3.4
-    docs: https://hadoop.apache.org/docs/r3.3.4
-    api: https://hadoop.apache.org/docs/r3.3.4/api
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L126
+    version: 3.4.1
+    docs: https://hadoop.apache.org/docs/r3.4.1
+    api: https://hadoop.apache.org/docs/r3.4.1/api
   hive:
-    version: 2.3.9
-    api: https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r2.3.9/api
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L135
+    version: 2.3.10
+    api: https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r2.3.10/api
   java:
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L117
     version: 17
     api: https://docs.oracle.com/en/java/javase/17/docs/api/java.base
     spec: https://docs.oracle.com/javase/specs/jls/se17/html
   kafka:
-    version: 3.4.1
-    api: https://kafka.apache.org/34/javadoc
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L137C20-L137C25
+    version: 3.9.0
+    api: https://kafka.apache.org/39/javadoc
   scala:
-    version: 2.13.8
-    api: https://www.scala-lang.org/api/2.13.8
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L173
+    version: 2.13.16
+    api: https://www.scala-lang.org/api/2.13.16
     docs: https://docs.scala-lang.org/
     spec: https://www.scala-lang.org/files/archive/spec/2.13
   social:
@@ -134,8 +139,9 @@ extra:
     - icon: fontawesome/brands/medium
       link: https://jaceklaskowski.medium.com
   spark:
-    version: 3.5.4
-    github: https://github.com/apache/spark/tree/v3.5.4
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L29
+    version: 4.0.0
+    github: https://github.com/apache/spark/tree/v4.0.0
     jira: https://issues.apache.org/jira/browse
   # https://squidfunk.github.io/mkdocs-material/reference/#setting-the-page-status
   status:

requirements.txt

@@ -1,6 +1,6 @@
 git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git
 mkdocs-minify-plugin>=0.7.2
-mkdocs-git-revision-date-localized-plugin>=1.2.2
+mkdocs-git-revision-date-localized-plugin>=1.4.7
 mkdocs-git-revision-date-plugin>=0.3.2
 mkdocs-awesome-pages-plugin>=2.9.2
 mkdocs-redirects>=1.2.1