Vars x-axis alignment with null values

[wrowlands3]

``I want two plots to be aligned when using vars, and a nice smooth line using window averages. But the raw data comes from an unreliable entity (ble, returns 'unavailable' often). When I use an averaging filter after resample it removes these null values and changes its alignment on the xaxis, with the effect more pronounced historically:

The heating fill in both of the below is correct, but the temperature line has moved significantly

An average filter with a window size of 1 before the resample fixes this, and I can apply another averaging filter with a higher number after for a smoother line. This makes historical data align well despite the accumulation of null values:

However, this solution seems to mess up the alignment a lot more for the current day of the latest data, unless I zoom/pan further out and back in.

Without first averaging (accurate):

With first averaging filter (accurate historically, but alignment jumps around with slight dragging of the latest data), again the vars.heating is the fill and it remains in place but the vars temperature moves:

I think the heating vars has the most reliable x-axis, as I have a map_y filter to address the null values; I also use the heating vars as the map_x xs[i] because the data does not extend that far back, and it keeps the heating vars ys[i] plotted to the correct time when I view before the entity was created and I want to view the temperature earlier than that.

entities:
  - entity: sensor.heat_trv_position
    internal: true
    line:
      width: 0
    fill: none
    filters:
      - resample: 5m
      - map_y: 'y === null ? 0 : ys[i]'             # seems to make the vars.heating.xs more reliable 
      - sliding_window_moving_average:
          window_size: 2
          extended: true
      - store_var: heating
  - entity: sensor.ble_temperature_hygro_5
    internal: true
    filters:
      - sliding_window_moving_average:       # improves overall accuracy, particularly historical, but gives the issue in the nearer term
          window_size: 1
      - resample: 5m
      - sliding_window_moving_average:
          window_size: 10
          extended: true
      - store_var: temperature
  - entity: sensor.ble_temperature_hygro_5
    line:
      width: 5
      shape: spline
    fill: none
    filters:
      - resample: 5m
      - map_y: vars.temperature.ys[i]
      - map_x: vars.heating.xs[i]      # this seems to have no effect on this query, I use it when viewing the temperature previous to the creation of the heating entity
      - sliding_window_moving_average:
          window_size: 10
          extended: true
  - entity: sensor.ble_temperature_hygro_5
    filters:
      - resample: 5m
      - map_y: 'vars.heating.ys[i] > "71" ? vars.temperature.ys[i] : 0'
      - map_x: vars.heating.xs[i]  # if I map this to vars.temperature.xs, then view before the heating entity was created then vars.heating.ys maps onto the current view and becomes completely unaligned
  - entity: sensor.ble_temperature_hygro_5
    filters:
      - resample: 5m
      - map_y: 'vars.heating.ys[i] > "51" ? vars.temperature.ys[i] : 0'
      - map_x: vars.heating.xs[i]
  - entity: sensor.ble_temperature_hygro_5
    filters:
      - resample: 5m
      - map_y: 'vars.heating.ys[i] > "21" ? vars.temperature.ys[i] : 0'
      - map_x: vars.heating.xs[i]
layout:
  yaxis:
    title:
      text: Temperature °C
      font:
        color: rgba(0, 153, 125,1)
    side: right
    range:
      - 14
      - 24
  paper_bgcolor: rgba(255,255,255,1)
  plot_bgcolor: rgba(0,0,0,0)
defaults:
  entity:
    unit_of_measurement: ' '
    line:
      width: 0
      color: rgba(0, 153, 125,1)
      shape: linear
    fill: tozeroy
    fillcolor: rgba(0, 153, 125,0.25)
hours_to_show: 18
plotly_raw_config: true

I'm close to making this stable, it only seems to be the current view can get misaligned, while zooming out and back in (without clicking reset view) to whatever timeframe I want it is perfect!

[dbuezas]

Yes moving window averaging adds a "delay" to the data of half the window size. You can compensate for that by adding centered: true to the filter options (next to extended)

As you noted, null values affect the moving window. That is because (as you already realized), the first thing that filter does is removing the "unavailable" values, and the window is not time based but element based instead.
Your trick of using resample is the way to fix that, just add the centered: true option and you should be fine :)

[dbuezas]

Try with:

- force_numeric
- resample
- sliding_window_moving_average

Alternatively you could use

- filter: y !== null
- resample
- sliding_window_moving_average

[dbuezas]

Although the averaging filter should do that already 🤔
What exactly are you going for? Can you make a scribble of what you expect the result to look like?

[wrowlands3]

Thanks for the suggestions, I managed to make it work with this:

    filters:
      - resample: 5m
      - map_y: 'y === null ? ys[ys.length-1] : ys[i]'  # assigns values to the null points based on the previous data point
      - sliding_window_moving_average:
          window_size: 10           #not needed prior to resample anymore to bridge null points, only for smoothing after

This essentially completes what I've been trying to achieve, thanks for all your pointers, I'll detail what I've got now in my original show and tell

[dbuezas]

- map_y: 'y === null ? ys[ys.length-1] : ys[i]'

This assigns the last datapoint to all the nulls, are you sure you want that?
I think you'll be better off with

- filter: y !== null
- resample
- sliding_window_moving_average

This removes all null values and then refills the gaps with previous non-null datapoints (resample)

[wrowlands3]

You are absolutely right, thanks for the correction!