How to run __.<task> from build.mill

[eikek]

Hi,

I'm new to mill (0.12.14) and trying to convert some of my sbt projects to mill. I've come to a few points I cannot solve. One is, I'd like to know how to do the cli __.test from within my own build task. In sbt, I used to have a ci alias/command that simply runs several things in sequence. This is then called from github actions and elsewhere. In sbt it would be very simple to run first format checks, then all tests, then perhaps a publishLocal using addCommandAlias.

I tried with a root module that contains my other modules and I would like to define a single command that runs some tasks on all child modules. But I'm stuck trying to run tasks on my "child modules", like all tests or all checkformats:

object `package` extends RootModule {

  def ci() = Task.Command {
     // run __.checkStyle
     // run __.test
  }


  object module1 extends ScalaModule with StyleModule {}
  object module2 extends ScalaModule with StyleModule {
     object test extends ScalaTests with TestModule.MUnit {
       // ...
     }
  }
}

I know from the cli I can use "command queries" like __.test, but how can I do this from inside build.mill?

I was playing with the millModuleDirectChildren to obtain all test modules and call Task.sequence on them. But this fails badly with weird exceptions (Duplicated item inserted into OrderedSet: java.nio.file.FileAlreadyExistsException). I then tried with mill.main.RunScript.evaluateNamed(ev, Agg(test.test())) for example, but can't succeed with this either (I had to make the RootModule a ScalaModule which it is not).

Thank you

[lefou]

If you already have a handle to all modules, using Task.traverse should be enough to get these tasks handled as prerequisites, and executed if not up-to-date.

def myTestModules: Seq[TestModule]

def ci() = Task.Command {
  // run __.test
  Task.traverse(myTestModules)(_.testForked())()
}

If you don't have the modules, you can use an evaluator command to find these modules. (Please note, that this API will slightly change in Mill 1.0.)

import mill.eval.{Evaluator, Terminal}
import mill.resolve.{Resolve, SelectMode}

def ci(evaluator: Evaluator) = Task.Command {
  Resolve.Tasks.resolve(
    evaluator.rootModule,
    Seq("__.testForked"),
    SelectMode.Separated
  ) match {
    case Left(err) => Task.fail(err)
    case Right(resolved) =>
      Task.sequence(resolved)()
  }
}

(I haven't actually run these examples, so there might be typos or errors in it)

[eikek]

Hi @lefou thank you very much for your quick reply!!

Unfortunately, I'm still not much further. I tried your code and what is written at the documentation you linked to. The snippet above (slightly modified to fit to 0.12.14) fails with

Target#apply() call cannot use `value resolved` defined within the Task{...} block

So I moved the resolving outside the Task.Command and arrived here:

  def ci(ev: Evaluator) = {
    val tests: List[mill.define.NamedTask[Any]] = Resolve.Tasks
      .resolve(
        ev.rootModule,
        Seq("__.test"),
        SelectMode.Separated
      )
      .fold(sys.error, identity)
    println(s"Found ${tests}")
    Task.Command {
      Task.sequence(tests)()
      ()
    }
  }

This compiles and runs without errors, but there are no tests run. I noticed that a NamedTask has a t: Task[…] field, when trying this (tests.map(_.t)) the exception about a duplicate item in an ordered set appears.

java.lang.Exception: Duplicated item inserted into OrderedSet: java.nio.file.FileAlreadyExistsException: a/b/out/ci.dest/testargs

I also tried with naming my modules directly:

  def ci2()= Task.Command {
    Task.traverse(List(server.test, common.test, store.test))(_.test())()
  }

It shows the same effect: no errors from mill, but no tests run either.

[lefou]

You can run mill show __.sources and mill show __.allSourceFiles and validate it matches with your real layout. If you have a sbt or Maven layout (e.g. src/main/scala instead of just src), you might want to use SbtModule in addition to or instead of ScalaModule.

[eikek]

I did that already and the folder looked ok. But I'll do more experiments, clearly I have some misconfiguration somewhere. It is probably due to silly me trying too many things at once. I also wanted to split the build into multiple files and got some issues with that as well, perhaps this is now mixing other parts up.

[eikek]

Oh no, I'm sorry. The source folder is wrong indeed! There is a folder in the middle missing, but my eyes were only checking the last part 🤦🏼

I think I got it sorted out, all entirely my fault. Just for the curious: I was playing around with the root module and wanted to have "global" tasks for ci etc as explained above. I was checking some discussions and came to this snippet:

def cc(ev: mill.eval.Evaluator): Command[Unit] = Task.Command(exclusive = true) {
  mill.main.RunScript.evaluateNamed(ev, Agg(clean(ev)))
  mill.main.RunScript.evaluateNamed(ev, Agg(compile))
  mill.main.RunScript.evaluateNamed(ev, Agg(test.test()))
  ()
}

The root module doesn't have a compile task. So I tried mixing in my module trait into the RootModule as well - just for experimenting, it felt very wrong… :-)

object `package` extends RootModule with MyModule {
  ...
}

And in my module trait, I'm redefining the millSourcePath so that each module is inside a modules folder.

  def millSourcePath = {
    val pre = super.millSourcePath
    pre / os.up / "modules" / pre.last
  }

This is working fine, if my modules are all defined outside the root module. But when inside this root module, first the root modules source path is changed and based on that change the source paths of my modules… argh. So when outside the root module, it is ok and, of course, when removing the mixin.

So the mill ci task is working! The tests are executed.

As a feedback from a very new user: Maybe it's just me, of course, I find defining tasks like the ci example a normal thing :-) I'm surprised that it is rather verbose to do in mill. I have already another use case for myself: I like to start a database before running all the tests, then run all the tests and then shut the database down. The tests itself can assume a running db, connect ando create random databases. For this I need to control running "all tests in the build" somehow.

Thank you very much for your time!

[lefou]

Glad, you figured it out.

Overlooking some minor difference in source paths happens to all of us. That's why I suggested to run mill show __.allSourceFiles. Although it can be hard to spot differences in source paths, it's rather easy to spot if there are no files at all.

[eikek]

Ah I understand, yes good point!