[BUG]: Cannot join to external master node

[SWATOPLUS]

I want to use Azure DataBricks for local integration testing. I have write some test for this.

        [Test]
        public void Test()
        {
            var spark = SparkSession.Builder().Master("--conn-string-here--").GetOrCreate();
            var row1 = new GenericRow(new object[] { 42, 4.3 });
            var row2 = new GenericRow(new object[] { 82, 4.8 });
            var dataFrame = spark.CreateDataFrame(new[] { row1, row2 }, new StructType(new[]
            {
                new StructField("Int", new IntegerType()),
                new StructField("Double", new DoubleType()),
            }));

            var result = dataFrame.Select("Double").ToLocalIterator().SelectMany(x => x.Values).ToArray();

            result.Should().BeEquivalentTo(new object[] { 4.3, 4.8 });
        }

If I don't set SparkSession.Builder().Master() property test works well with local spark-submit started in debug mode. If I set some some spark url like "spark://4.2.2.2:8795" or just wrong address like "#" the builder throws network exception with localhost ip address (127.0.0.1)

Looks like this property do nothing and should be fixed.

By the way I also tried to connect to Azure DataBricks I get connection string for it from WebUI it also redirects to localhost.

[Niharikadutta]

@SWATOPLUS Are you using Spark standalone cluster deployment? If yes, can you confirm where your Spark Master is running? That URL is the URL you would need to set at SparkSession.Builder().Master("spark-master-url").