panic: unsupported for aggregate min: *reads.stringMultiShardArrayCursor

[ttftw]

I have asked for help in Slack and will follow up with customer support, but I thought I would also post this here.

I am currently using influxdb cloud as our primary data store. All data is coming in from an mqtt server. I recently spun up a local influxdb instance in a docker container and am sending a copy of the data to a this new local server for testing. The local server adds a few new tags, but besides that, everything else should be the same. Same bucket name, same data structure, etc. I'm doing this so I can just drop in this new server into existing Grafana dashboards and test some things before we push changes that go to the production server.

When I run the same queries from the cloud against the local server, I have some that cause influxdb to panic.

from(bucket: "bucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "Sample")
  |> filter(fn: (r) => r["serial"] == "xxx.xxx")
  |> filter(fn: (r) => r["channel"] == "Cellular")
  |> aggregateWindow(every: v.windowPeriod, fn: min, createEmpty: false)

This runs fine on the cloud server, but when I run it locally, I get

panic: unsupported for aggregate min: *reads.stringMultiShardArrayCursor

and if I comment out the aggregateWindow, then it returns the data as I'd expect. One thing to note is when I run this query, the _value column data type is a double in the cloud and a long on the local instance, but besides that, the tables look identical when I comment out the aggregateWindow.

Oddly enough, I just noticed that if I change the query from

|> filter(fn: (r) => r["channel"] == "Cellular")

to

|> filter(fn: (r) => r["_field"] == "Status.Cellular")

which targets the same data rows, but a different way, then I do not get this panic and the data returns as expected with the aggregateWindow.

Any insight as to why this query would be successful in the cloud version but panic in the OSS version will be much appreciated.

InfluxDB v2.7.11
Server: fbf5d4a

Logs:

influxdb-1 | ts=2025-03-14T15:39:37.538278Z lvl=info msg="Execute source panic" log_id=0vHGw81W000 service=storage-reads error="panic: unsupported for aggregate min: *reads.stringMultiShardArrayCursor" stacktrace="goroutine 207758 [running]:\nruntime/debug.Stack()\n\t/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/influxdata/flux/execute.(*executionState).recover(0xc002241170)\n\t/go/pkg/mod/github.com/influxdata/flux@v0.195.2/execute/recover.go:32 +0x1fd\npanic({0x7f33d4a24b40?, 0xc0036f9650?})\n\t/go/src/runtime/panic.go:770 +0x132\ngithub.com/influxdata/influxdb/v2/storage/reads.newWindowMinArrayCursor({0x7f33d502de40?, 0xc00735d850?}, {{0x0, 0x2540be400, 0x0}, {0x0, 0x2540be400, 0x0}, 0x0, 0x0, ...})\n\t/root/project/storage/reads/array_cursor.gen.go:158 +0x40c\ngithub.com/influxdata/influxdb/v2/storage/reads.newWindowAggregateArrayCursor({0xc0069e5dc0?, 0x9?}, 0x9?, {{0x0, 0x2540be400, 0x0}, {0x0, 0x2540be400, 0x0}, 0x0, ...}, ...)\n\t/root/project/storage/reads/array_cursor.go:43 +0x9c\ngithub.com/influxdata/influxdb/v2/storage/reads.(*windowAggregateResultSet).createCursor(0xc0069e3770, {{0x0, 0x0, 0x0}, {0x7f3384608495, 0x6, 0x3f7b6b}, {0xc002071340, 0x7, 0x7}, ...})\n\t/root/project/storage/reads/aggregate_resultset.go:131 +0x2a6\ngithub.com/influxdata/influxdb/v2/storage/reads.(*windowAggregateResultSet).Next(0xc0069e3770)\n\t/root/project/storage/reads/aggregate_resultset.go:84 +0x138\ngithub.com/influxdata/influxdb/v2/storage/flux.(*windowAggregateIterator).handleRead(0xc00295a100, 0x7f33d5035218?, {0x7f33d5048520, 0xc0069e3770})\n\t/root/project/storage/flux/reader.go:742 +0x3aa\ngithub.com/influxdata/influxdb/v2/storage/flux.(*windowAggregateIterator).Do(0xc00295a100, 0xc0016b3ae0)\n\t/root/project/storage/flux/reader.go:688 +0x365\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).processTables(0xc002070f20, {0x7f33d5035218, 0xc0036fe090}, {0x7f33d5020118, 0xc00295a100}, 0x182cb58e09149c46)\n\t/root/project/query/stdlib/influxdata/influxdb/source.go:69 +0x9a\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*readWindowAggregateSource).run(0xc002070f20, {0x7f33d5035218, 0xc0036fe090})\n\t/root/project/query/stdlib/influxdata/influxdb/source.go:303 +0x106\ngithub.com/influxdata/influxdb/v2/query/stdlib/influxdata/influxdb.(*Source).Run(0xc002070f20, {0x7f33d5035218, 0xc0036fe090})\n\t/root/project/query/stdlib/influxdata/influxdb/source.go:50 +0xa3\ngithub.com/influxdata/flux/execute.(*executionState).do.func2({0x7f33d5036d70, 0xc002070f20})\n\t/go/pkg/mod/github.com/influxdata/flux@v0.195.2/execute/executor.go:535 +0x375\ncreated by github.com/influxdata/flux/execute.(*executionState).do in goroutine 234\n\t/go/pkg/mod/github.com/influxdata/flux@v0.195.2/execute/executor.go:515 +0xf8\n"

[devanbenz]

@ttftw I've opened up a PR to remove the erroneous panic. That being said I'll also need to take a look at the query to see why its being transformed to use the improper cursor type. Thank you for the detailed bug report. Going to continue looking in to it today.

[devanbenz]

Also if possible but would you be willing to send me over some line protocol for the schema you're using so I can write data to a local influxdb to help with debugging? I'll try my best to reproduce some mock data with the query you've provided in the mean time.

[ttftw]

@devanbenz Here is a sample of data that generated the error above:

Sample,alarm=False,channel=Cellular,client=None,driver=Status,project=None,serial=PV112,site=Demo Status.Cellular=-17i 1742420358

[devanbenz]

@ttftw thank you - another question: How exactly are you running the flux query? Are you using the chronograf UI or running it some other way?

[ttftw]

I've tried and confirmed the error from Grafana dashboards/Explore and the Data Explorer tool in Influxdb

[devanbenz]

I've been attempting to reproduce this issue locally without much luck. I assume that this data was line protocol you dumped from a TSM file?

Sample,alarm=False,channel=Cellular,client=None,driver=Status,project=None,serial=PV112,site=Demo Status.Cellular=-17i 1742420358

Just ensuring I am reading it correctly. I assume that your tags set is the following:

alarm=False,channel=Cellular,client=None,driver=Status,project=None,serial=PV112,site=Demo

Can you verify that I have tags right? I may request that you give me a clone of some mock data in the form of a TSM file you're seeing the issue on if possible. I would like to try and reproduce this issue myself.

[devanbenz]

Okay so I was able to reproduce this issue. I basically wrote some points in line protocol using a script. I modified my points so that they were a string type for _value which is just the field value as outlined by our documentation here: https://docs.influxdata.com/influxdb/cloud/reference/key-concepts/data-elements/#fields. Obviously I only have 1 field being channel but I wonder if you have multiple fields and theres a conflict between them somehow... My suspicion is that you have some sneaky _value 's locally that are of string type?

Documents/InfluxData/issue_26142 via 🐍 v3.10.12 on ☁  dbenz@influxdata.com took 18s
❯ python3 repro_lp.py
Data saved to influxdb_data.txt

Sample of generated data:
Sample,alarm=true,channel=WiFi,client=Client1,driver=Status,project=Project3,serial=PV112,site=Production WiFi="bar" 1742324017
Sample,alarm=true,channel=Cellular,client=None,driver=Sensor,project=None,serial=PV112,site=Production Cellular="baz" 1742324077
Sample,alarm=false,channel=Cellular,client=Client3,driver=Monitor,project=None,serial=PV112,site=Staging Cellular="bar" 1742324137
Sample,alarm=false,channel=Ethernet,client=Client2,driver=Control,project=Project1,serial=PV112,site=Development Ethernet="baz" 1742324198
Sample,alarm=true,channel=Bluetooth,client=Client1,driver=Status,project=Project1,serial=PV112,site=Production Bluetooth="foo" 1742324258

I've gone ahead and added a PR for this case where instead of panic'ing we will error. #26165

On your local instance where you're seeing the issue could you run your flux query without the aggregation?

from(bucket: "bucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "Sample")
  |> filter(fn: (r) => r["serial"] == "xxx.xxx")
  |> filter(fn: (r) => r["channel"] == "Cellular")

And send over a screen grab from the Simple Table graph like so?

I have a feeling that somehow the query is attempting to aggregate on string values instead of numerical values.