Verbosity System Description

[jClugstor]

The current way to specify verbosity for solvers is through the Bool keyword argument verbose, which just causes the solver to print out more information if it's true. We can have a lot more flexibility by adding an AbstractVerbosity abstract type and concrete subtypes for every SciMLProblem type.

For example we'll have a BVPVerbosity <: AbstractVerbosity that will have toggle-able Bool fields for everything that we would want to print during a BVP solve.

We can have constructors so that BVPVerbosity(true) has every field true, BVPVerbosity(false) has every field false, and BVPVerbosity(nothing) is the default which has some things true and other things false.

This will allow the user to have much greater control over what is printed during the solution of any problem.

[jClugstor]

Related to SciML/BoundaryValueDiffEq.jl#290 (comment)

[ChrisRackauckas]

Some things to read, especially from R's global warning system, which I think has done the best for user-level toggling of messaging:

• https://stackoverflow.com/questions/11239428/how-to-change-warning-setting
• https://stat.ethz.ch/R-manual/R-devel/library/base/html/warning.html
• https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/warnings
• https://adv-r.hadley.nz/conditions.html

The idea is that we want to get something with similar-ish functionality, but without relying on globals, which is not thread safe and has performance issues, instead allowing this to be at a function level and uniform across SciML. So let's keep that goal in mind.

The basic kwargs on everything should be:

• verbose: highest level toggle on all printing and warning defaults. Defaults to Verbosity.Default, Allowed: None, Default, All
• error_control: toggles handling on error control and adaptivity checking algorithms, such as convergence issues and correctness guarantees (e.x. dt < dtmin). Defaults to Verbosity.Warn. Allowed Verbosity: None, Warn, Error.
• performance: toggles handling on performance issues, such as mismatched input/output types in out of place functions and other known performance traps. Defaults to Verbosity.Warn. Allowed Verbosity: None, Warn, Error.
• numerical: toggles handling of potential numerical issues, such as large condition numbers and detection of other potential floating point issues. Defaults to Verbosity.None. Allowed: None, Edge, All.

Edge = only prints the info on edge cases, like high condition numbers and changes to QR pivoting.
Info = prints every step. It's a debugging mode. This could also have things like solver_choice, where it can print on things like "I switched which ODE solver I'm using here".

Then each of these should be handling individual settings like convergence, dtmin, condition, etc. which are flipped by the higher commands, so error_control = Verbosity.Error sets all of the error control pieces like convergence = Verbosity.Error

verbose is then treated as a default where others refine. All just turns on all printing available, so then every condition number prints out etc, and then you can condition = Verbosity.None to override that.

The normal usage should match what we have right now, i.e. that "most" things try not to print but warnings of error control do print to let the user know why the solver exited early. Then we expect users to do things like:

verbose = ODEVerbosity(verbose = Verbosity.None) # I want no printing ever, quickest way to write that
verbose = ODEVerbosity(verbose = Verbosity.All) # I want all printing and I'm just going to copy paste it into an issue and Chris why the solver is failing
verbose = ODEVerbosity(error_control = Verbosity.Error) # I want error messages instead of warnings if the solver fails
verbose = ODEVerbosity(performance= Verbosity.Warn) # I want warnings for solver exiting but also if something is slow 
verbose = ODEVerbosity(performance= Verbosity.Edge) # I'm scared that my equation is hard and want more information

Those are probably the most common scenarios right there.

In addition we want to start controlling logging locations. Right now we assume everything goes to the REPL. But if we start to print out giant lists of condition numbers, usually I don't Revise that to my REPL but slap that into a file. So it would be nice to give users easy way to do this. In fact, think of some of these options as "wouldn't it be great if the user gave me everything in a bug report so I didn't have to run their code myself?" The point of this is to make that as dead simple as possible, so users know how to fix things better but also give us better bug reports.

Again, something someone like @oscardssmith or I would do is add printing statements into OrdinaryDiffEq to do things like, print if the solver switched from stiff to non-stiff, and print the time stamp of when that happened, so if verbose = ODEVerbosity(performance= Verbosity.All) did exactly that and slapped it into a file, we could just ask the user to run that report and we could diagnose what's going on from our phones without having to run the code ourselves. Quicker debugging and user feedback.

So for logging control we probably want some stuff like warn_location, info_location, etc. where it defaults to REPL, but then has an option for FILE, where then a filename is provided. This would be nice for example so that the exit warnings still go to the REPL, but you can have the extra verbose information printing out to a file.

There's probably many other ideas to go on this. @ranocha @devmotion @oscardssmith @AayushSabharwal @BenChung @Vaibhavdixit02 @avik-pal @baggepinnen @bradcarman @asinghvi17 @SebastianM-C probably have feedback, along with maybe @LilithHafner who has done some with Julia Base and likely has some suggests of things we should use to accomplish this.

[AayushSabharwal]

Then each of these should be handling individual settings like convergence, dtmin, condition, etc. which are flipped by the higher commands, so error_control = Verbosity.Error sets all of the error control pieces like convergence = Verbosity.Error

Could we make the hierarchy more explicit? So we have a struct ErrorControlVerbosity <: AbstractVerbosity and in the top level ODEVerbosity struct the field is error_control::Union{Verbosity, ErrorControlVerbosity}. The user either sets error_control = Verbosity.Warn or error_control = ErrorControlVerbosity(; dtmin = Verbosity.Warn). We can define convert(::Type{<:AbstractVerbosity}, ::Verbosity) to automatically convert Verbosity.Warn into an ErrorControlVerbosity. This could also enable a nice way to say "Everything but X", by doing

error_control = ErrorControlVerbosity(Verbosity.None)
error_control.dtmin = Verbosity.Warn

[SebastianM-C]

One aspect that I want to mention on this is that the Logging stdlib & LoggingExtras have several of the features that we want, such as

• printing to file
• various log levels
• adding metadata to logs that can be used for filtering

One issue though is that if you have a log that is disabled, such as the default @debug behavior, that can still lead to a performance hit since the codegen still contains logging code, so adding this kind of logging would not be zero cost. It would be very nice if base julia could have some options for statically defining log levels or something similar.

[ranocha]

I agree with the last part - this should not come at the price of making something like solving many small ODEs on a GPU in parallel terribly slow.

[ChrisRackauckas]

Yes, it should all compile out when the logging isn't used. That's pretty essential and would break static compilation if that isn't setup. So that's definitely in the requirements.

[jClugstor]

If Enums are a problem we can use Moshi.jl, something like this:

@data Verbosity begin
    None
    Edge
    Info
    All
    Default
end

which will keep the namespacing of enums (like Verbosity.None), but really are just singleton types are not just singleton types like I thought they were, so maybe this won't work like I though it will. Anyway, we already depend on Moshi in SciMLBase, and the macros should be pretty stable, so I think it's reasonable to use for this if it can work.

Then we can have for example something like this:

abstract type ProblemVerbositySpecifier end

struct LinearVerbosity <: ProblemVerbositySpecifier
    error_control::LinearErrorVerbosity
    performance::LinearPerformanceVerbosity
    numerical::LinearNumericalVerbosity
end


struct LinearErrorControlVerbosity
    #fields that set verbosity for specific things
    something::Verbosity

    #fields that set where info goes and where warnings go
    info_location
    warn_location
end

function LinearErrorControlVerbosity(::Verbosity.None)
    #set everything to Verbosity.None
end

function LinearErrorControlVerbosity(::Verbosity.Default)
    #set everything to defaults
end

# etc. etc.

struct LinearPerformanceVerbosity
    something_else::Verbosity

    info_location
    warn_location
end

#constructors

struct LinearNumericalVerbosity
    large_condition::Verbosity

    info_location
    warn_location
end

#constructors

So each problem will have a verbosity specifier associated with it, and each verbosity specifier will have
$(Problem)ErrorControlVerbosity, $(Problem)PerformanceVerbosity, $(Problem)NumericalVerbosity, as fields. Then each of those will have fields for toggling the specific things. It's a lot of types, but each problem type will have different error control things it can toggle and different performance things it can toggle etc. so I think it makes sense. By the way, the types will probably have to have fields that cover all of the possible things you would want to print, even though it might not be applicable to every solver. For example if using an implicit solver for an ODE you might want to see warnings related to the Jacobian calculation. That doesn't apply to explicit solvers but we probably want to be able to toggle it anyway.

Also, each lowest level struct will have info_location and warn_location. This will allow for the most granularity, you might want to see your numerical warnings in a log, but your performance info in the REPL. Of course we should be able to set these at a high level as well.

@AayushSabharwal I like the idea, but I'm not sure I would want e.g. error_control to ever just be a Verbosity. I think it would be nice if users could always go in to the ODEVerbosity and look at exactly what the verbosity settings are for every toggle. If the error_control field was a Verbosity.Default you wouldn't be able to look at the details of what exactly is set.

[termi-official]

I think there are two sides for this system to consider. One side is indeed providing users with direct information with useful messages about what is happening, as already discussed here, as well as tracing the code paths taken to debug e.g. integrators. The other side is having callbacks which can be customized by users, i.e. solver monitors to print out or store custom information about different stages of the simulation process. A common monitor example in PDE-world is to print out the intermediate solutions for each iteration in Newton solvers to debug issues with e.g. boundary conditions of the model. I think it should also be possible to compile these callbacks into no-ops when passing NoMonitor-type structs into the inner loops.

[jClugstor]

@ChrisRackauckas Just to be clear, for the lowest level toggles like convergence = Verbosity.Error, that should mean if there's a convergence issue we should give errors, correct? And then we want to be able to do things like convergence = Verbosity.Warn where these will be warnings instead, or convergence = Verbosity.Info where these will be @info messages? Then we can have convergence = Verbosity.None to turn off all printing for convergence messages?

In other words the lowest level toggles should set whether or not specific things are printed out, and if they are, what level the messages are?