Filters

[dbuezas]

What's Changed

• Filters suport
• Deprecated lambdas - WARNING there may be some incompatibilities

Instructions in the readme of the draft branch here

Full Changelog: v1.8.4...v2.0.0-beta

---

This discussion was created from the release Filters.

[dbuezas]

Warning: beta version, some filters may change before production release

Examples

Celcious to farenheit

- entity: sensor.wintergarten_clima_temperature
  unit_of_measurement: °F
  filters: # °F = °C×(9/5)+32
    - multiply: 1.8
    - add: 32

alternatively,

- entity: sensor.wintergarten_clima_temperature
  unit_of_measurement: °F
  filters: # °F = °C×(9/5)+32
    - map_y_numbers: y * 9/5 + 32

Energy from power

- entity: sensor.fridge_power
  filters:
    - integrate:
        unit: h # resulting unit_of_measurement will be W/h

Using state attributes

- entity: climate.loungetrv_climate
  attribute: current_temperature # an attribute must be set to ensure attributes are fetched.
  filters:
    - map_y_numbers: state.state === "heat" ? state.attributes.current_temperature : 0

more in the branch's readme here!

[dbuezas]

Thanks! I'm really happy with how the filters worked out.
The issue is likely just yaml not liking the colon. Here are some variations:

type: custom:plotly-graph-dev
title: binary sensor
entities:
  - entity: binary_sensor.flur_motion
    filters:
      - map_y: |- 
          y === "on" ? 1 : 0
  - entity: binary_sensor.flur_motion
    filters:
      - map_y: y === "on" ? 3:2
  - entity: binary_sensor.flur_motion
    filters:
      - map_y: 'y === "on" ? 5 : 4' # yaml doesn't like the colon when written like this and unescaped  

---

BTW, next time please paste your yaml and/or a screenshot. It's easy to know what went wrong :)
Did you pick this from an example? If so, feel free to correct it by clicking on the pencil at the top right of the readme, then editing, and then "create branch and start a pull request"

[mwrben]

Thank you. Your suggestion of how to write it into yaml worked.

On a separate (minor) note, I think the yaml example in the readme needs re-aligning. Presumably the line starting filters: needs to be
re-aligned further along.

type: custom:plotly-graph
entities:

• entity: sensor.temperature_in_celsius
  filters:

  The filters below will only be applied to numeric values. Missing (unavailable) and non-numerics will be left untouched

  • add: 5 # adds 5 to each datapoint

[dbuezas]

Can you edit your message like this:
```yaml
like: this
```

It's hard to read otherwise:)

[dbuezas]

Also, please try editing the readme yourself. You can do it from the browser. I'll review it make any necessary correction and merge it :)

[dbuezas]

Oh, I see, the example in the readme was missing the outer quotes. I fixed it now.

[dbuezas]

Small example of how simple it will be to do complex operations

type: custom:plotly-graph
title: Smoothing
entities:
  - entity: sensor.openweathermap_wind_speed
  - entity: sensor.openweathermap_wind_speed
    name: Moving window average
    filters:
      - sliding_window_moving_average:
          extended: true
          window_size: 20
  - entity: sensor.openweathermap_wind_speed
    name: Moving window median
    filters:
      - median:
          extended: true
          window_size: 20

hours_to_show: 240

[dbuezas]

type: custom:plotly-graph
title: Forecast
offset: 1.5d
entities:
  - entity: sensor.openweathermap_rain
    legendgroup: 1
    name: Precipitation
  - entity: weather.openweathermap
    legendgroup: 1
    showlegend: false
    name: Precipitation
    line:
      dash: dot
      shape: spline
    unit_of_measurement: mm
    filters:
      - fn: |-
          ({ meta }) => ({
            xs: meta.forecast.map(({ datetime }) => new Date(datetime)),
            ys: meta.forecast.map(({ precipitation }) => precipitation),
          })
  - entity: sensor.openweathermap_temperature
    legendgroup: 2
    name: Temperature
  - entity: weather.openweathermap
    legendgroup: 2
    showlegend: false
    name: Temperature
    line:
      dash: dot
      shape: spline
    unit_of_measurement: °C
    filters:
      - fn: |-
          ({ meta }) => ({
            xs: meta.forecast.map(({ datetime }) => new Date(datetime)),
            ys: meta.forecast.map(({ temperature }) => temperature),
          })
  - entity: sensor.openweathermap_wind_speed
    legendgroup: 3
    name: Wind speed
  - entity: weather.openweathermap
    legendgroup: 3
    showlegend: false
    name: Wind speed
    line:
      dash: dot
      shape: spline
    unit_of_measurement: m/s
    filters:
      - fn: |-
          ({ meta }) => ({
            xs: meta.forecast.map(({ datetime }) => new Date(datetime)),
            ys: meta.forecast.map(({ wind_speed }) => wind_speed),
          })
color_scheme:
  - '#1f77b4'
  - '#1f77b4'
  - '#d62728'
  - '#d62728'
  - '#7fc97f'
  - '#7fc97f'
hours_to_show: 72

[b3nn0]

Just upgraded to the new version and started to replace my Lambdas.
AWESOME! it's so much simpler. THANK YOU!

I only failed replacing one of them. Now I wonder if I'm missing something, or if it might be a nice feature suggestion:
With Statistics and auto-ranging, the X axis interval can be quite flexible, showing 5 minute to monthly intervals depending on zoom.
I have a water meter (and gas meter) delivering an absolute value, and I'm creating a bar chart by derivative of the statistics sensor.
To do so, I simply subtract the "previous" value from each value. And due to auto-ranging, I get a "5 minute water usage" bar chart or a "monthly water usage bar chart".

However, the derivative-function of the filters seems to require a time-unit, so I can only create fixed bar charts (e.g. water usage per hour).
Would it be able to add a "derivative from last value" option or something, so this becomes possible?
Or is there another way to achieve something like this?

Yaml:

type: custom:plotly-graph
title: Wasserverbrauch
entities:
  - entity: sensor.wasserzahler
    name: Wasserverbrauch
    statistic: sum
    unit_of_measurement: L
    period: auto
    showlegend: false
    show_value: false
    type: bar
    texttemplate: '%{y}'
    lambda: |-
      (ys, xs) => {
        const res = []
        for (let i = 1; i < ys.length; i++) {
          res.push((ys[i] - ys[i-1]) * 1000)
        }
        xs.shift()
        return {x: xs, y: res }
      }
defaults:
  yaxes:
    fixedrange: true
hours_to_show: 48
refresh_interval: 10

Additionally, the unit seems to not work correctly for me in the derivate filter..
If I just put derivate: in there, I get a hourly derivate graph (L/h).
However, if I put anything else there (including unit: h), I get no graph at all:

EDIT: Oops: If I indent the "unit:" one less, I do get a graph, but it's always hourly..?

[b3nn0]

Hmm, not sure what the scaling factor means in that context, or what difference it would make (per day derivative is the same, no matter if I have 1d samples or 1h samples, no?)

Anyway, this one:

       - resample: 1d
       - derivate:
            unit: d

produces the same result for me: a completely empty graph.
Removing the "unit: d" creates a daily "L/h" graph - just the same as without "resample: 1d". E.g. if I use 240L/d, it will show "10.0", because 24*10L = 240L.

And it also doesn't really solve the nice behaviour of period: auto statistics, with "dynamic resampling duration depending on zoom".

I'm fine with keeping it a lambda for now, so at least the period: auto part isn't too important, just nice to have.
But I just can't seem to get any thing to work when specifying unit: ...?

[b3nn0]

Ah.. that's why:
Error: Filter 'derivat must be [force_numeric,add,multiply,calibrate_linear,derivate,integrate,sliding_window_moving_average,median,exponential_moving_average,map_y_numbers,map_y,map_x,resample,store_var,fn,filter]

EDIT: never mind.. that was just while typing.. it shows no error in the console.

[dbuezas]

not sure what the scaling factor means in that context

It is exactly what you wrote below: I use 240L/d, it will show "10.0", because 24*10L = 240L.

But I just can't seem to get any thing to work when specifying unit: ...?

Mmmm you may have found a bug. Would you rise an issue so I don't forget to look into it?

[dbuezas]

Ohhhh nevermind, the readme is wrong. It should be derivate: 1d without the unit subobject.

[dbuezas]

I updated the readme, thanks for reporting this! I think I misunderstood you the first time, I thought you expected to have one bar a day with derivate: 1d.
Regarding dynamic resampling, I could add it if you convince me of its utility :). Feel free to open a feature request and put some screenshots and scribbles of what you intend to do with it.

[kaijk]

I'm trying to move kWh bar charts for daily, and hourly, and stacked daily and hourly (by time of day) energy from lambda to filters. No joy. The data come from Watt readings. What I'm trying to accomplish is to have bar graphs begin at midnight 'x' days or months, or some hours ago and continue through the current partial period, with multiple traces stacked by separate on-peak, off-peak accumulations (for which I have separate Watt sensors).

Here is my simplest setup, trying to achieve one column per day with a single series, no stacking. In HA I use the Rhimann integration with the left method against this same sensor to get energy:

 - type: custom:plotly-graph
    entities:
      - entity: sensor.power_meter_total_watts
        type: bar
        filters:
          - integrate: d
    hours_to_show: 72
    refresh_interval: 30
    title: Energy Consumption Daily
    layout:
      yaxis:
        title: Wh per Day

This produces three days of continuously increasing interval data, not binned by day, but in bar format by native intervals that looks more like a fill.

Removing the type: bar just removes the "fill" appearance.

If I use integrate: 1d as noted above for derivate, I get a graph full of NANs. (BTW, I like the idea of xd to specify a multi-day period over which to integrate (or 0d for just the current day) ).

This is the most complex example I'm trying to reproduce that uses lambda. It happens to be hourly, but I also want monthly and daily by Bus and by TOD, stacked. Computation efficiency is important to not overwhelm my RasPi 4, if possible.

[dbuezas]

integrate: 1d is not a valid parameter. It only supports raw time units (ms, s, h, d, ...), because the time unit is not designed to be a reset interval, but just the time unit, and something like "2d" would result in a weird unit_of_measurement with a constant: kW2d
Meaning, there is no built in filter available at the moment for what you need.

With that being said, I think it is a cool feature to add.

Would you open a feature request so I can track it and discuss the details with you? We can discuss the API there (I am thinking about either a new filter, or extra parameters to integrate, like reset_every: 1d.

[kaijk]

Will do. Thank you for making all this happen!

[thecem]

Is there a way to have a stacked bar beside a bar in a line chart?
Or other way around, is there a config to group only certain entitys beside a group of stacked in one chart?

[dbuezas]

Oh, that I don't know. You'll have to google or search here: https://plotly.com/javascript/reference/

[kaijk]

Add stackgroup: <stringa> to those entities you want stacked in group A, stackgroup: <stringb> to those entities you want stacked in group B, and leave stackgroup missing from those you do not want stacked. See https://plotly.com/javascript/reference/scatter/#scatter-stackgroup

[thecem]

thanks a lot, I tried but even the easiest bar charts with two entitys are not shown as a "group" but overlayed:
Some hints whats wrong?

[thecem]

var trace1 = {
  x: ['giraffes', 'orangutans', 'monkeys'],
  y: [20, 14, 23],
  name: 'SF Zoo',
  type: 'bar'
};

var trace2 = {
  x: ['giraffes', 'orangutans', 'monkeys'],
  y: [12, 18, 29],
  name: 'LA Zoo',
  type: 'bar'
};

var data = [trace2, trace1];

var layout = {barmode: 'group'};

Plotly.newPlot('myDiv', data, layout);

Whre could i adress the order : data = [trace2, trace1]; ?

[thecem]

I think "barmode: group" needs more data to work, as addressed here: #184 (comment)

[kaijk]

How might I convert this lambda to one of the new filters?

lambda: |-
       (ys, xs) => (
           {
               x: xs.slice(-1).map(x=> x && x),
               y: ys.slice(-1).map(y=> y && y),
            }
         ) 

Do I read this correctly as set x for all xs to lag(1)(x), else (x) (and for y as well)? I'd still like to know how to do it using filters

[dbuezas]

I don't understand what your objective is. The function above just removes everything but the very last value. x && x has no effect.
[0,1,2,3].slice(-1) results in [3]