.NET for Apache Spark accessing Event Hubs

[michaelwinick]

I am trying to write a simple .NET for Apache Spark application that continuously reads events from a Azure Event Hub and writes events to a local csv file. I've based this example from the code I got from gist previously submitted by a Microsoft developer, Jose Perez Rodriguez. https://gist.github.com/joperezr/de0c963213c69b870f2e65b92c5406a4. I can't seem to get past this error:

[2021-09-12T17:05:08.4617669Z] [5CD034PMMX] [Error] [JvmBridge] java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.eventhubs.EventHubsSourceProvider could not be instantiated

Here are the relevant libraries I am using:

• Java version: 1.8.0_301
• Spark version: 3.0.1: Scala version 2.12.10, Java 1.8.0_301
• .NET for Apache Spark version: 2.0

I want to leverage the Spark eco system in the .NET world, so I am very excited, but a bit frustrated sifting thru the tons of output.

mySparkApp.zip
spark_submit.txt
error_output.txt