Bug: Default value provider is not called for required inherited options of subcommands.

[garretwilson]

I'm seeing some behavior I don't understand.

I have an application class Foo like this:

@Command(name = "foo", subcommands = {Bar.class})
public class FooCli {

  private String widget = nll;

  @Option(names = "--widget", scope = ScopeType.INHERIT)
  protected setWidget(String widget) {
    this.widget = widget.
  }


}

Then I have a Bar subcommand class:

@Command(name = "bar")
public class Bar {

  @Command(name = "baz");
  public void baz() {
  }

}

In the main Foo class I set up a default value provider:

  commandLine.setDefaultValueProvider(new CommandLine.IDefaultValueProvider() {
    @Override
    public String defaultValue(final ArgSpec argSpec) throws Exception {
      return findDefaultCliArg(argSpec).orElse(null);
    }
  });

That winds its way down to this method:

  protected Optional<String> findDefaultCliOption(final OptionSpec optionSpec) {
    System.out.println("looking for default for command `%s` option `%s`: ".formatted(optionSpec.command().qualifiedName(), optionSpec.longestName()));
    return Optional.empty(); //TODO implement
  }

This is just to see what is going on.

If I run foo bar baz with no --widget option, I see something like this in the output:

looking for default for command `foo` option `--widget`

That makes sense, because --widget is defined at the foo application command level, and it's trying to find the value, since I didn't supply one.

And this is exactly what I want, because if I'm doing a default lookup of an inherited option, I don't want to check for a thousand different commands—I just want to look for the option on the command it was defined.

But what if I make --widget required?

  @Option(names = "--widget", required = true, scope = ScopeType.INHERIT)

Suddenly the output changes!

looking for default for command `foo bar baz` option `--widget`

So the default lookup passes me a fully defined command foo bar baz CommandSpec if the --widget option is required, but only the top-level foo CommandSpec if --widget is not required!

Why the difference? Shouldn't the default lookup pass the provider exactly the same information regardless of whether the option is required?

[garretwilson]

Interestingly (and this is somewhat beside the point), I removed the scope = ScopeType.INHERIT from Foo.setWidget(…), and picocli still tried to look up the default for it even when I invoked foo bar baz. I had to reread #649, but I think the reason is that options are automatically inherited by class subcommands (even without ScopeType.INHERIT), but not method subcommands (which was the motivation for #649). That's fine—I've just never used class-based subcommands before, so I had forgotten this detail and was putting ScopeType.INHERIT everywhere.

So it seems that this is another oddity: if I leave off scope = ScopeType.INHERIT, the option is not inherited to the subcommand(s), but picocli still tries to look up a default value for that option. I don't know that this will cause any problems in my app, but I don't see how this is correct. If it is impossible for a user to specify the non-inherited option on a subcommand, why would picocli try to look up a default? I don't even know what that would mean—to have a default value for an option that doesn't really exist.

[garretwilson]

I found out what is going on, and the situation is arguably worse.

What happens when I make a global command required (or even add a new global command that is required), picocli doesn't do any default lookups at all!! If I make --widget required but don't pass a widget, then commandLine.execute(args) will fail with a message like this:

Missing required option: '--widget=<widget>'

This comes from inside internalHandleParseException(…). As soon as it prints that error, it then prints the usage:

            writer.println(colorScheme.errorText(ex.getMessage()));
            if (!UnmatchedArgumentException.printSuggestions(ex, writer)) {
                ex.getCommandLine().usage(writer, colorScheme);
            }

It is inside ex.getCommandLine().usage(writer, colorScheme) that picocli apparently tries to do default lookups. Here is where the weird stuff happens:

• picocli calls the default value provider to try to get default values, regardless of whether the option was optional or required.
• picocli passes the full qualified command (e.g. foo bar baz) as the context to the default value provider. This is where I was seeing this being called with a different context, which is why I filed this ticket. But it was happening during usage printing, after failure had already occurred because of the missing required option.
• The default value provider seems to be called multiple times. Perhaps the usage printing is calling it for foo, then for foo bar, and then for foo bar baz (just a guess)—nevertheless it sends the entire qualified foo bar baz as the command spec.

So picocli is doing a lot of unexpected things inside the usage printing that occurs after a missing required option is encountered. That explains the behavior I saw. But it reveals a bigger problem, which is really a severe bug: if I mark an option as required, I can no longer use the default value provider lookup mechanism!

Let me walk through the reasoning here. Let's say I add the following:

@Option(names = "--widget", required = true, scope = ScopeType.INHERIT)

If I don't provide a --widget option, then picocli will complain and fail the application. That is expected and desired. But what if I provide a default value?

@Option(names = "--widget", required = true, defaultValue = "gingerbread" scope = ScopeType.INHERIT)

If I don't provide a --widget option on the command line, picocli will use the default value "gingerbread". Again that is completely expected and desired.

But what if I have a default value provider installed? Isn't the default value provider supposed to provide a dynamic, customized approach to provide the default value—semantically equivalent to the defaultValue annotation attribute? Shouldn't a default value provider be able to provider a default value for a required option, equivalent to a literal defaultValue just in a more customizable fashion?

It's possible I'm missing something here, but it sure looks like picocli is ignoring the registered default value provider for required options.

[garretwilson]

The properties file default value provider is a good example of the bug here. If a user puts a default value for --widget in the properties file (I haven't looked closely at what the syntax is, as I'm writing my own similar mechanism), then picocli would still fail and say effectively, "you didn't provide a --widget". But this makes no sense to the user, because they did provide a --widget—in the properties file!

[remkop]

That sounds like a bug.
Can you provide a JUnit test that demonstrates the issue?

[garretwilson]

It turns out this is a tricky one, because it only shows up when you try to invoke a subcommand; the required global option default value provider is not called if the global optional is marked as required. If you just call the root command, the default value provider is called. That's probably why this never surfaced before.

Here is a JUnit example using Hamcrest:

import static org.hamcrest.MatcherAssert.*;
import static org.hamcrest.Matchers.*;

import java.util.*;

import org.junit.jupiter.api.*;

import picocli.CommandLine;
import picocli.CommandLine.*;
import picocli.CommandLine.Model.*;

public class PicocliIssue2443Test {

  @Test
  void verifyPicoCliCallsDefaultValueProviderForOptionalGlobalOption() {
    final PicocliIssue2443App picocliIssue2443App = new PicocliIssue2443App();
    final CommandLine commandLine = new CommandLine(picocliIssue2443App);
    final Set<String> argsCallingDefaultValueProvider = new HashSet<>();
    commandLine.setDefaultValueProvider(new IDefaultValueProvider() {
      @Override
      public String defaultValue(final ArgSpec argSpec) throws Exception {
        argsCallingDefaultValueProvider.add(((OptionSpec)argSpec).longestName());
        return "foobar";
      }
    });
    commandLine.parseArgs("do", "something", "--required-global-code", "xyz");
    assertThat(argsCallingDefaultValueProvider, contains("--optional-global-code"));
  }

  @Test
  void verifyPicoCliCallsDefaultValueProviderForRequiredGlobalOption() {
    final PicocliIssue2443App picocliIssue2443App = new PicocliIssue2443App();
    final CommandLine commandLine = new CommandLine(picocliIssue2443App);
    final Set<String> argsCallingDefaultValueProvider = new HashSet<>();
    commandLine.setDefaultValueProvider(new IDefaultValueProvider() {
      @Override
      public String defaultValue(final ArgSpec argSpec) throws Exception {
        argsCallingDefaultValueProvider.add(((OptionSpec)argSpec).longestName());
        return "foobar";
      }
    });
    commandLine.parseArgs("do", "something", "--optional-global-code", "abc");
    assertThat(argsCallingDefaultValueProvider, contains("--required-global-code"));
  }

  @Command(name = "issue2443", subcommands = {PicocliIssue2443App.DoCommand.class})
  class PicocliIssue2443App implements Runnable {

    @Option(names = "--optional-global-code", scope = ScopeType.INHERIT)
    String optionalGlobalCode;

    @Option(names = "--required-global-code", required = true, scope = ScopeType.INHERIT)
    String requiredGlobalCode;

    @Override
    public void run() {
    }

    @Command(name = "do", subcommands = {DoCommand.SomethingCommand.class})
    static class DoCommand implements Runnable {
      @Override
      public void run() {
      }

      @Command(name = "something")
      static class SomethingCommand implements Runnable {
        @Override
        public void run() {
        }
      }
    }
  }

}

The verifyPicoCliCallsDefaultValueProviderForRequiredGlobalOption() test will fail with this stack trace (some values redacted):

picocli.CommandLine$MissingParameterException: Missing required option: '--required-global-code=<requiredGlobalCode>'
	at picocli.CommandLine$MissingParameterException.create(CommandLine.java:18714)
	at picocli.CommandLine$MissingParameterException.access$21300(CommandLine.java:18702)
	at picocli.CommandLine$Interpreter.validateConstraints(CommandLine.java:13680)
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13643)
	at picocli.CommandLine$Interpreter.processSubcommand(CommandLine.java:13938)
	at picocli.CommandLine$Interpreter.processArguments(CommandLine.java:13834)
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13620)
	at picocli.CommandLine$Interpreter.processSubcommand(CommandLine.java:13938)
	at picocli.CommandLine$Interpreter.processArguments(CommandLine.java:13834)
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13620)
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13588)
	at picocli.CommandLine$Interpreter.parse(CommandLine.java:13483)
	at picocli.CommandLine.parseArgs(CommandLine.java:1552)
	at ….Picocli2443Test.verifyPicoCliCallsDefaultValueProviderForRequiredGlobalOption(Picocli2443Test.java:…)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)