Getting started to build/test the code locally

[minnieshi]

on mac os, installed mill; and run the compile by
mill __.compile
then I tried to run any of the below without success

mill spark-excel[2.12.20,2.4.8].test
./mill __.test
mill spark-excel[2.12.20,3.2.4].test.testOnly dev.mauch.spark.excel.PlainNumberReadSuite
mill __.test

i used sdkman, java version currently is below

openjdk version "17.0.6" 2023-01-17 LTS

example failure output

[122/122] ============================== spark-excel[2.12.20,2.4.8].test ============================== 32s
1 tasks failed
spark-excel[2.12.20,2.4.8].test.test 88 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files must infer correct schema with inferSchema=true
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files with input_file_name
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read whole folder with partition
and 83 more ...

another example:

$ mill spark-excel[2.12.20,3.2.4].test.test dev.mauch.spark.excel.PlainNumberReadSuite
[122] java.lang.IllegalArgumentException: Argument unrecognized by ScalaTest's Runner: dev.mauch.spark.excel.PlainNumberReadSuite
[122/122] spark-excel[2.12.20,3.2.4].test.test
[122] at org.scalatest.tools.ArgsParser$.parseArgs(ArgsParser.scala:451)
[122] at org.scalatest.tools.Framework.runner(Framework.scala:983)
[122] at mill.testrunner.TestRunnerUtils$.getTestTasks(TestRunnerUtils.scala:116)
[122] at mill.testrunner.TestRunnerUtils$.runTestFramework0(TestRunnerUtils.scala:204)
[122] at mill.testrunner.TestRunnerMain0$.main0(TestRunnerMain0.scala:38)
[122] at mill.testrunner.TestRunnerMain0.main0(TestRunnerMain0.scala)
[122] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[122] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[122] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[122] at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[122] at mill.testrunner.entrypoint.TestRunnerMain.main(TestRunnerMain.java:42)
[122] java.lang.IllegalArgumentException: Argument unrecognized by ScalaTest's Runner: dev.mauch.spark.excel.PlainNumberReadSuite
[122/122] ============================== spark-excel[2.12.20,3.2.4].test.test dev.mauch.spark.excel.PlainNumberReadSuite ============================== 1s
1 tasks failed
spark-excel[2.12.20,3.2.4].test.test Test reporting Failed: /Users/blahblah/code/spark-excel/out/spark-excel/2.12.20/3.2.4/test/test.dest/out.json does not exist

the failure when i had plainly run

mill __.test

[1256] Run completed in 25 seconds, 948 milliseconds.
[1256] Total number of tests run: 111
[1256] Suites: completed 21, aborted 0
[1256] Tests: succeeded 23, failed 88, canceled 7, ignored 0, pending 0
[1256] *** 88 TESTS FAILED ***
[1262/1262] ============================== __.test ============================== 35s
10 tasks failed
spark-excel[2.12.20,2.4.8].test.test 88 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files must infer correct schema with inferSchema=true
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read multiple files with input_file_name
dev.mauch.spark.excel.v2.GlobPartitionAndFileNameSuite read whole folder with partition
and 83 more ...
spark-excel[2.12.20,3.0.3].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.0.3/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.2.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.2.4/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.3.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.3.4/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.4.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.4.4/test/test.dest/out.json does not exist
spark-excel[2.12.20,3.5.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.12.20/3.5.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.2.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.2.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.3.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.3.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.4.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.4.4/test/test.dest/out.json does not exist
spark-excel[2.13.15,3.5.4].test.test Test reporting Failed: /Users/min.shi/code/spark-excel/out/spark-excel/2.13.15/3.5.4/test/test.dest/out.json does not exist
MacBookPro:spark-excel min.shi$

[nightscape]

Hi @minnieshi, can you give JDK 11 a try?
Only the newer Spark versions (not sure which ones exactly) are compatible with JDK 17.
There can still be a low number of test errors which I haven't exactly figured out, but it should be just two or three.

[minnieshi]

i did this

sdk use java 11.0.25-amzn
mill spark-excel[2.12.20,3.2.4].test

[122] *** 4 TESTS FAILED ***
[122/122] ============================== spark-excel[2.12.20,3.2.4].test ============================== 92s
1 tasks failed
spark-excel[2.12.20,3.2.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false

will report result for
mill __.test soon

[minnieshi]

the
mill __.test output is below
@nightscape

I believe the test output is no longer related to my local environment setup.
if you agree, then I will open/fix the issue with a separate topic/title.

[1262/1262] ============================== __.test ============================== 427s
10 tasks failed
spark-excel[2.12.20,3.0.3].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.12.20,3.1.3].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.12.20,3.2.4].test.test 5 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.IntegrationSuite with maxRowsInMemory = None; maxByteArraySize = Some(100000000) reads files without headers correctly
spark-excel[2.12.20,3.3.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.12.20,3.4.4].test.test 5 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.IntegrationSuite with maxRowsInMemory = None; maxByteArraySize = Some(100000000) handles multi-line column headers correctly
spark-excel[2.12.20,3.5.4].test.test 5 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.IntegrationSuite with maxRowsInMemory = Some(1); maxByteArraySize = None reads blank cells as null and empty string cells as ""
spark-excel[2.13.15,3.2.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.13.15,3.3.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.13.15,3.4.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false
spark-excel[2.13.15,3.5.4].test.test 4 tests failed:
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.PlainNumberReadSuite spark-excel should read numbers in excel general number format when usePlainNumberFormat=false and inferSchema=false
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=true
dev.mauch.spark.excel.v2.PlainNumberReadSuite excel general number format when usePlainNumberFormat=false and inferSchema=false

[nightscape]

Yes, these look like real test failures. They are different than the ones that fail for me sometimes. For me it's just dev.mauch.spark.excel.IntegrationSuite, whereas for you there's also the dev.mauch.spark.excel.v2.PlainNumberReadSuite.

[minnieshi]

Now, new challenges have arrived with the intention of debugging/checking the failed tests in the IDE. What are the suggestions for the next step?

My IDE is IntelliJ.

Imported the project as a BSP project:

File > New > Project from Existing Sources.
Select the project directory.
Choose the BSP option for import

the import auto-generated the file .bsp/mill-bsp.json with below content
{ "name": "mill-bsp", "argv": [ "/Users/min.shi/code/spark-excel/mill", "--bsp", "--disable-ticker", "--color", "false", "--jobs", "1" ], "millVersion": "0.12.5", "bspVersion": "2.2.0-M2", "languages": [ "java", "scala", "kotlin" ] }

Clicking the 'bsp' reload button does not have issues (i.e., there are no errors in the build window; please see 2nd screenshot), but IntelliJ does not understand the build.mill file. (see 1st screenshot of the problem window or the 'red lines' in the Editor

below is the problem window

below is the build window, no issues ($HOMELibrary/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/lihaoyi/mill-contrib-sonatypecentral_2.13/0.12.5/mill-contrib-sonatypecentral_2.13-0.12.5-sources.jar is downloaded)

[nightscape]

Hi @minnieshi, thanks for the detailed report 👍🏻
I never got Intellij to properly highlight the build.mill.
Intellij also sometimes has issues with normal non-build-related code.
As long as the build and compilation itself works I wouldn't worry.
Do you have any issues with compiling/running/debugging, or is it just the build.mill highlighting?