Add proposal for sensitive data exclusion in Ballerina logging

[TharmiganK]

Purpose

$Subject

Discussion issue #1376

[ayeshLK]

What is the usecase for users to enable sensitive data logging ?

[TharmiganK]

This is useful in testing in development time where we need to verify some sensitive data. Also the proposed method has its own performance tradeoff(getting the mask string representation requires to check for the annotation which can affect the performance in runtime), and if the developer avoided printing any sensitive data by themself, then they don't want this performance drop.

[ayeshLK]

This is useful in testing in development time where we need to verify some sensitive data.

IMO, it's likely that sensitive data would be stored in a datastore, so viewing it in logs might not be a common use case during dev testing.

Also the proposed method has its own performance tradeoff(getting the mask string representation requires to check for the annotation which can affect the performance in runtime), and if the developer avoided printing any sensitive data by themself, then they don't want this performance drop.

Agree that there would be a performance hit with the annotation processing. Did we consider any alternative approaches (maybe from Java logging frameworks) to solve this problem ? Maybe a pattern-matching based approach ? [1][2]

[1] - https://is.docs.wso2.com/en/7.0.0/deploy/monitor/log4j-mask-sensitive-information-in-logs/
[2] - https://www.baeldung.com/logback-mask-sensitive-data

[TharmiganK]

The alternative section covers this where we can pass the masking function to the custom logger. But that is not intutive enough. Since Ballerina is type oriented, following approach seems to be suitable for me.

[TharmiganK]

Had a call, and agreed to do a comprehensive performance test once this is implemented. Based on the results we may have to disable it as default

[daneshk]

I think having a configuration that enables sensible data masking and make it disabled by default is the right way to go. MI has the same experience. https://mi.docs.wso2.com/en/latest/administer/logging-and-monitoring/logging/masking-sensitive-information-in-logs/

[TharmiganK]

Yes, we will proceed like that

Btw the perf results: #1376 (comment)

[gimantha]

@TharmiganK While keeping BI in mind, if a low code developer (Who has no idea about the Ballerina concepts), try to use this feature but unaware of Ballerina's type system and directly use JSON or XML or any other text format, how would he apply these EXCLUDE and REPLACEMENT strategies via either annotation or function pointer?

[TharmiganK]

If the developers are using plain JSON or XML values then it is not possible to use this feature since the current proposed approach is coupled with the type system.

In that case, having a pattern matching approach (suggested by @ayeshLK) might work. For custom loggers developer has to implement the mask function (refer to alternative section) and for root logger, it can be based on pattern matching and the the regex patterns to mask sensitive data can be configured via the Config.toml.

@daneshk @ThisaruGuruge Appreciate your input on this

(Btw IMO, not knowing the Ballerina's type system might greatly affect the experience in low code. They cannot ensure type-safety and moreover they cannot use features that are coupled with type system - constraint validation)

[TharmiganK]

Had a call, and agreed to go ahead with the annotation based approach. Discuss about BI support for annotation to support annotation based features