DEBUG logs not showing up (need assistance)

[lymanaukuleles]

I have these logging configuration settings in my elastalert.yaml :

logging:
 version: 1
 incremental: false
 disable_existing_loggers: false
 formatters:
   logline:
     format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'

 handlers:
   console:
     class: logging.StreamHandler
     formatter: logline
     level: DEBUG
     stream: ext://sys.stderr

 loggers:
   elastalert:
     level: DEBUG
     handlers: []
     propagate: true

   elasticsearch:
     level: DEBUG
     handlers: []
     propagate: true

   elasticsearch.trace:
     level: DEBUG
     handlers: []
     propagate: true

   '':  # root logger
     level: DEBUG
     handlers:
       - console
     propagate: false

This is my docker logs :

INFO:elastalert:Sleeping for 59.999674 seconds
INFO:elastalert:Queried rule High Priority Rules File from 2022-02-16 08:46 UTC to 2022-02-16 09:01 UTC: 7 / 7 hits
INFO:elastalert:Alert for High Priority Rules File at 2022-02-16T09:00:38.275Z:
INFO:elastalert:High Priority IPS
Details of the event:
- Syslog Priority Level: <MISSING VALUE>
- Category of Issue: <MISSING VALUE>
- Firewall WAN IP: <MISSING VALUE>
- Complete msg field: <MISSING VALUE>


INFO:elastalert:Alert for High Priority Rules File at 2022-02-16T09:00:38.275Z:
INFO:elastalert:High Priority IPS
Details of the event:
- Syslog Priority Level: <MISSING VALUE>
- Category of Issue: <MISSING VALUE>
- Firewall WAN IP: <MISSING VALUE>
- Complete msg field: <MISSING VALUE>


INFO:elastalert:Alert for High Priority Rules File at 2022-02-16T09:01:02.110Z:
INFO:elastalert:High Priority IPS
Details of the event:
- Syslog Priority Level: <MISSING VALUE>
- Category of Issue: <MISSING VALUE>
- Firewall WAN IP: <MISSING VALUE>
- Complete msg field: <MISSING VALUE>


INFO:elastalert:Skipping writing to ES: {'rule_name': 'High Priority Rules File', 'endtime': '2022-02-16T09:01:43.781091Z', 'starttime': '2022-02-16T08:46:43.781091Z', 'matches': 3, 'hits': 7, '@timestamp': '2022-02-16T09:01:43.814889Z', 'time_taken': 0.033766746520996094}
INFO:elastalert:Ran High Priority Rules File from 2022-02-16 08:46 UTC to 2022-02-16 09:01 UTC: 7 query hits (4 already seen), 3 matches, 0 alerts sent
INFO:elastalert:High Priority Rules File range 900
INFO:elastalert:Queried rule Lockout Rules File from 2022-02-16 08:46 UTC to 2022-02-16 09:01 UTC: 0 / 0 hits
INFO:elastalert:Skipping writing to ES: {'rule_name': 'Lockout Rules File', 'endtime': '2022-02-16T09:01:55.620304Z', 'starttime': '2022-02-16T08:46:55.620304Z', 'matches': 0, 'hits': 0, '@timestamp': '2022-02-16T09:01:55.666296Z', 'time_taken': 0.04596209526062012}
INFO:elastalert:Ran Lockout Rules File from 2022-02-16 08:46 UTC to 2022-02-16 09:01 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Lockout Rules File range 900
INFO:elastalert:Background configuration change check run at 2022-02-16 09:02 UTC
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-02-16 09:02 UTC
INFO:elastalert(teeth)isabled rules are: []
INFO:elastalert:Sleeping for 59.999658 seconds

But I haven't been able to get a DEBUG log with the elasticsearch fields.

[jertel]

Use docker exec to get inside the running ElastAlert 2 container, and then less the config file to see if your logging changes are actually being mounted into the container.

[lymanaukuleles]

Because initially I was having a yaml.parser.ParserError and indenting the settings solved the error. However now I can pull the settings back to where they normally would be and see if it changes anything.

[jertel]

Indents mean everything in yaml, so what you pasted above would cause the logging changes to not be applied.

[lymanaukuleles]

I redid it so the yaml is like this:

# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
logging:
  version: 1
  incremental: false
  disable_existing_loggers: false
  formatters:
    logline:
      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'

handlers:
   console:
     class: logging.StreamHandler
     formatter: logline
     level: DEBUG
     stream: ext://sys.stderr

loggers:
   elastalert:
     level: DEBUG
     handlers: []
     propagate: true

elasticsearch:
    level: DEBUG
    handlers: []
    propagate: true

elasticsearch.trace:
    level: DEBUG
    handlers: []
    propagate: true

'':  # root logger
   level: DEBUG
   handlers:
     - console
   propagate: false

However I am getting an error in my docker logs :

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/elastalert/create_index.py", line 201, in main
    data = yaml.load(config_file, Loader=yaml.FullLoader)
  File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
  File "/usr/local/lib/python3.10/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python3.10/site-packages/yaml/composer.py", line 39, in get_single_node
    if not self.check_event(StreamEndEvent):
  File "/usr/local/lib/python3.10/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.10/site-packages/yaml/parser.py", line 171, in parse_document_start
    raise ParserError(None, None,
yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>'
  in "/opt/elastalert/config.yaml", line 86, column 1

Can you help? Note that line 86 corresponds to logging: in the config. So for example its like:

86. logging:
87.   version: 1
88.   incremental: false
etc....

[jertel]

Your indentation of the subordinate blocks is now wrong. Follow the example in this example, as mentioned in the docs. Once you uncomment out the logging section the logging: line should have no ident, but the subordinate blocks, such as handlers: and loggers: need to be indented.

[lymanaukuleles]

Your indentation of the subordinate blocks is now wrong. Follow the example in this example, as mentioned in the docs. Once you uncomment out the logging section the logging: line should have no ident, but the subordinate blocks, such as handlers: and loggers: need to be indented.

Thank you. Getting the indentation correct managed to get the DEBUG logs to appear.