Stream names are not friendly with logging systems (mlflow, wandb, etc.)

[tjhunter]

Is your feature request related to a problem? Please describe.

Currently, we allow descriptive stream names such as NPP, ATMS following a known convention in the satellite world. See this example. However, logging systems only want simple names for metrics such as a.b.c. where a is a-zA-Z0-9-_ . It could be more but I would not push because it would break pandas column selection.

As a result, we cannot really have per-stream metrics, which would be useful for example to track how many samples are being ingested per stream. In this case it would be streams.NPP, ATMS.count_samples which would cause issues with WandB or MLFlow.

I see 3 ways forward:

• magic: conversion of NPP, ATMS to camelCase NppAtms for example
• convention: only allow names that follow the convention above and return error instead
• extra info: when defining a stream name, require a user to set an stream_id as a unique machine-ready identifier (for example NPP_ATMS)

In any case, I believe this name should be a-zA-Z0-9_ (no _ to prevent weird issues with databases or pandas)

https://stackoverflow.com/questions/47964380/pandas-dataframe-column-naming-conventions

Do you have any thoughts on that?

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Organisation

No response

[clessig]

Can't we use "NPP, ATMS" as the user friendly identifier and convert automagically to the machine friendly form (like NppAtms) that is typically hidden from the user?

[tjhunter]

That sounds good to me. Explicit is always better than Implicit, but I do not have strong opinions for this naming convention.

[clessig]

That sounds good to me. Explicit is always better than Implicit, but I do not have strong opinions for this naming convention.

For the user format: for obs, "[PLATFORM, INSTRUMENT]" is standard. For the internal format, camel case with all offending chars parsed away sounds good.

[jpolz]

I'll start working on this

[jpolz]

The easy part: conversion to camelcase

import re

def camel_case(s, regex=r"[a-zA-Z0-9]+"):
    """
    Convert a string to camelCase. Characters to omit are all not in given regular expression.
    
    :param s: Input string
    :param regex: Regular expression defining valid characters in words
    :return: CamelCase version of the input string
    """
    # Split the string into words based on characters not in regular expression
    re_pattern = re.compile(regex)
    words = re_pattern.findall(s)

    if not words:
        return ""
    
    # Capitalize the first letter of each word
    camel_cased = ''.join(word.capitalize() for word in words)
    
    return camel_cased

camel_case("NPP, ATMS") gives NppAtms

@tjhunter Is retrieving the original string desired or will we just store a copy? And where would you do the conversion?

[clessig]

We should retain the human readable form (like "NPP, ATMS") if possible and additionally store the machine friendly version.

For the function documentation, can we use the agreed on style.

[Sindhu-Vasireddy]

assign me