Support JSON Logging capability

[donatelloOo]

Issue submitter TODO list

• I've searched for an already existing issues here
• I'm running a supported version of the application which is listed here and the feature is not present there

Is your proposal related to a problem?

It's not possible to produce logs with custom JSON format.
It would be great to allow smooth integration with cluster's logging stack.

Note: the JSON format must be configurable (so the standard logback JSON output is not enough)

• Using logback, the following dependency is missing to allow configuring JSON encoder:

  <dependency>
      <groupId>net.logstash.logback</groupId>
      <artifactId>logstash-logback-encoder</artifactId>
  </dependency>
  

  see https://springframework.guru/json-logging-with-spring-boot/

• Using log4j2, the whole log4j2 slf4j impl + json layout support is missing (maybe by choice):

    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-slf4j2-impl</artifactId>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-layout-template-json</artifactId>
    </dependency>
  

  https://logging.apache.org/log4j/2.x/manual/json-template-layout.html

Describe the feature you're interested in

Add Custom JSON logging capability using Logback or Log4j2.

Describe alternatives you've considered

No response

Version you're running

1.5.0 helmchart / 1.2.0 image

Additional context

the spring boot integration with log4j is very easy if specifying for instance:

  logging:
    config: "/kafka-ui/log4j2.yaml"

[github-actions]

Hi donatelloOo! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues.
Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.

[Haarolean]

This can be achieved via logging.structured.format.console=ecs property without any additional dependencies. But rn we have a logback config overriding this (logback-spring.xml), which prevents us from seeing the logs in a desired format. I'll take a look to see if we can nuke the file.

[donatelloOo]

could we override the logback-spring.xml by overriding the logging conf with logging.config="/path/to/logback-spring.xml" ?
let me try.

[donatelloOo]

According to the Spring Boot documentation, the JSON logging format can be customized (default is Elastic Common Schema), using this kind of configuration:

see https://docs.spring.io/spring-boot/reference/features/logging.html#features.logging.structured.customizing-json

yamlApplicationConfig:
  # ======================================================
  # Application Logging Configuration
  # Configure logging levels for specific packages. and output format.
  # ======================================================
  logging:
    config: "{{ .Values.extraConfigDir }}/logback-spring.xml"
    structured:
      format:
        console: ecs
      json:
        rename:
          # see Elastic Common Schema: https://www.elastic.co/guide/en/ecs/8.11/ecs-reference.html
          "[@timestamp]": log_timestamp
          log.logger: log_logger
          log.level: log_level
          message: log_message
          process.id: process_id
          process.name: process_name
          process.thread.id: thread_id
          process.thread.name: thread_name

## @param extraConfigDir Directory where extra config files ('files/configs/') will be mounted in the Kafka-UI container
extraConfigDir: "/kafka-ui/extra-config"

where the extracConfigDir is mounted with a configmap containing following data:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>${CONSOLE_LOG_THRESHOLD}</level>
        </filter>
        <!-- use Spring Boot StructuredLogEncoder -->
        <encoder class="org.springframework.boot.logging.logback.StructuredLogEncoder">
            <format>${CONSOLE_LOG_STRUCTURED_FORMAT}</format>
            <charset>${CONSOLE_LOG_CHARSET}</charset>
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>