Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 94 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,95 @@ The package is hosted with pypi (https://pypi.org/project/clp-logging/), so it
can be installed with `pip`:

`python3 -m pip install --upgrade clp-logging`

## Logging handlers

## Logger handlers
### ClpKeyValuePairStreamHandler

⭐ *New in v0.0.14*

This handler enables applications to write structured log events directly into CLP's key-value pair
(kv-pair) IR stream format. The handler accepts structured log events in the form of Python
dictionaries, where each dictionary entry must abide by the requirements detailed
[below](#key-value-pair-requirements). The handler will also automatically include certain
[metadata](#automatically-generated-kv-pairs), like the log event's level, with each log event.

> [!NOTE]
> Since this handler accepts structured log events, it doesn't support setting a
> [Formatter][py-logging-formatter] (because the log events don't need formatting into a string).

> [!WARNING]
> `ClpKeyValuePairStreamHandler` currently doesn't support
> [CLPLogLevelTimeout](#log-level-timeout-feature-clplogleveltimeout). This feature will be added in
> a future release.

#### Key-value pair requirements

`ClpKeyValuePairStreamHandler` requires kv-pairs abide by the following rules:

- Keys must be of type `str`.
- Values must be one of the following types:
- Primitives: `int`, `float`, `str`, `bool`, or `None`.
- Arrays (`list`), where each array:
- may contain primitive values, dictionaries, or nested arrays.
- can be empty.
- Dictionaries (`dict`), where each dictionary:
- must adhere to the aforementioned rules for keys and values.
- can be empty.

#### Automatically generated kv-pairs

In addition to the kv-pairs explicitly logged by the application, the handler will add kv-pairs,
like the log event's level, to each log event. We refer to the former as *user-generated* kv-pairs
and the latter as *auto-generated* kv-pairs.

> [!NOTE]
> The kv-pair IR stream format stores auto-generated kv-pairs separately from user-generated
> kv-pairs, so users don't need to worry about key collisions with the auto-generated keys.

The handler adds the following auto-generated kv-pairs to each log event:

| Key | Value type | Description |
|---------------------|------------|---------------------------------------------------|
| `timestamp` | `dict` | The log event's timestamp |
| - `unix_millisecs` | `int` | The timestamp as a Unix timestamp in milliseconds |
| - `utc_offset_secs` | `int` | The timestamp's UTC offset in seconds |
| `level` | `dict` | The log event's level |
| - `name` | `str` | The level's name |
| - `num` | `int` | The level's numeric value |
| `source_location` | `dict` | The source location of the logging statement |
| - `path` | `str` | The source location's path |
| - `line` | `int` | The source location's line number |

### Example: `ClpKeyValuePairStreamHandler`

```python
import logging
from pathlib import Path
from clp_logging.handlers import ClpKeyValuePairStreamHandler

clp_handler = ClpKeyValuePairStreamHandler(open(Path("example.clp.zst"), "wb"))
logger: logging.Logger = logging.getLogger(__name__)
logger.addHandler(clp_handler)

logger.info({
"message": "This is an example message",
"machine_info": {
"uid": 12345,
"ip": "127.0.0.1",
},
})
```

### Reading kv-pair IR streams

The following options are available for reading and deserializing kv-pair IR streams generated by
this handler:

- [clp-ffi-py][clp-ffi-py-pypi]: This library provides a [Deserializer][clp-ffi-py-deserializer-doc]
to access a kv-pair IR stream in Python. [This example][clp-ffi-py-deserializer-example]
illustrates its usage.
- [YScope Log Viewer][2]: This UI can be used to view kv-pair IR streams.

### CLPStreamHandler

Expand Down Expand Up @@ -267,3 +354,9 @@ word][7].
[7]: https://docformatter.readthedocs.io/en/latest/faq.html#interaction-with-black
[8]: https://beta.ruff.rs/docs/
[9]: https://github.com/y-scope/clp-ffi-py

[clp-ffi-py-deserializer-doc]: https://github.com/y-scope/clp-ffi-py?tab=readme-ov-file#example-code-using-deserializer-to-read-keyvaluepairlogevents-from-an-ir-stream
[clp-ffi-py-deserializer-example]: https://github.com/y-scope/clp-ffi-py?tab=readme-ov-file#example-code-using-deserializer-to-read-keyvaluepairlogevents-from-an-ir-stream
[clp-ffi-py-kv-pair-ir-stream]: https://github.com/y-scope/clp-ffi-py?tab=readme-ov-file#using-key-value-pair-ir-streams
[clp-ffi-py-pypi]: https://pypi.org/project/clp-ffi-py/
[py-logging-formatter]: https://docs.python.org/3/library/logging.html#logging.Formatter
Empty file added test.py
Empty file.
Loading