how to delete unused channel?

[artesby]

hello, i have just updated to milvus 2.5.6 and noticed this channel in new web ui

this channel belongs to collection that doesnt exist (probably, it was removed)
can I somehow drop this channel manually?

[yhmo]

Don't manually remove the channel.

If a collection is dropped, the dml channel should be removed accordingly by milvus.
You can use this tool to obverse the collections: https://milvus.io/docs/birdwatcher_install_guides.md
Download the executable binary according to your OS:

wget -O birdwatcher.tar.gz \
https://github.com/milvus-io/birdwatcher/releases/download/latest/birdwatcher_<os>_<arch>.tar.gz

And start the tool in terminal:

tar -xvzf birdwatcher.tar.gz
./birdwatcher

Connect to the etcd server which the milvus is using, and show collections:

Check the state of the collection whose ID is "xxxxx20460", to see if it is in "dropped" or "dropping" state.

If the collection is in "dropped" or "dropping" state but its channel is still displayed in WebUI, I think it could be a bug. To find the root case, we need the full log.

[xiaofan-luan]

this should be a old on 2.4.13 versions.

I guess the easiest way is to use birdwatcher and clean all information about this collection.

There should be some command related

[xiaofan-luan]

it seems the collection info is already purged to you only need to clean channel meta.

[artesby]

something like this?

remove channel --channel <id> --run --force

Usage:
   remove channel [flags]

Flags:
      --channel string   channel name to remove
      --force            force remove channel ignoring collection check
  -h, --help             help for channel
      --run              flags indicating whether to remove channel from meta

[xiaofan-luan]

Yep, try to use force.
then restart the node see if it works

[artesby]

Milvus(by-dev) > remove channel --channel by-dev/meta/channelwatch/184/by-dev-rootcoord-dml_3_453584804738020460v0 --run --force
Start to delete orphan watch channel info...

Milvus(by-dev) > show channel-watch
=============================
key: by-dev/meta/channelwatch/184/by-dev-rootcoord-dml_3_453584804738020460v0
Channel Name:by-dev-rootcoord-dml_3_453584804738020460v0 	 WatchState: ToWatch
Channel Watch start from: 2025-03-24 22:06:53 +0000, timeout at: 1970-01-01 00:00:00 +0000
Start Position ID: [], time: 1970-01-01 00:00:00 +0000
Unflushed segments: []
Flushed segments: []
Dropped segments: []
=============================

which pod should be restarted to take effect?

[xiaofan-luan]

you can probably use this command to fix channel watch
https://github.com/milvus-io/birdwatcher/blob/d2c2f60cae45108954ae791116586349aeb599b1/states/etcd/repair/channel_watched.go

and then restart coordinator

[artesby]

repair does not work

Milvus(by-dev) > repair channel-watch --collection 453584804738020460 
No empty schema watch info found

remove did the job

Milvus(by-dev) > remove channel --channel by-dev-rootcoord-dml_3_453584804738020460v0 --force --run
by-dev-rootcoord-dml_3_453584804738020460v0 selected as target channel, collection id: 453584804738020460
Start to delete orphan watch channel info...
audit delete by-dev/meta/channelwatch/184/by-dev-rootcoord-dml_3_453584804738020460v0
remove orphan channel by-dev/meta/channelwatch/184/by-dev-rootcoord-dml_3_453584804738020460v0 done

now show channels shows correctly 3 channels

--- Total Channels: 3

however, after restart of mixcoord webUI started to show this warning

and I still see warnings about this collection and channel in logs of mixcoord

kubectl -n milvus logs -f deploy/milvus-mixcoord | grep -e "dml_3" -e "453584804738020460" -e "not found"
{"level":"WARN","time":"2025/03/26 06:16:51.664 +00:00","caller":"rootcoord/meta_table.go:577","message":"not found collection","collectionID":453584804738020460}
{"level":"WARN","time":"2025/03/26 06:16:54.660 +00:00","caller":"rootcoord/meta_table.go:577","message":"not found collection","collectionID":453584804738020460}
{"level":"WARN","time":"2025/03/26 06:16:57.673 +00:00","caller":"rootcoord/meta_table.go:577","message":"not found collection","collectionID":453584804738020460}
{"level":"WARN","time":"2025/03/26 06:16:59.291 +00:00","caller":"datacoord/meta.go:780","message":"meta update: get segment failed - segment not found","segmentID":456878720580671445,"segment nil":true,"segment unhealthy":true}
{"level":"WARN","time":"2025/03/26 06:17:00.660 +00:00","caller":"rootcoord/meta_table.go:577","message":"not found collection","collectionID":453584804738020460}
{"level":"WARN","time":"2025/03/26 06:17:02.388 +00:00","caller":"datacoord/task_index.go:205","message":"failed to get dim from field type params","field type":"VarChar","error":"dim not found in params","errorVerbose":"dim not found in params\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/storage.GetDimFromParams\n  | \t/workspace/source/internal/storage/utils.go:233\n  | github.com/milvus-io/milvus/internal/datacoord.(*indexBuildTask).PreCheck\n  | \t/workspace/source/internal/datacoord/task_index.go:203\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).processInit\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:548\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).process\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:455\n  | github.com/milvus-io/milvus/internal/datacoord.(*taskScheduler).run.func1\n  | \t/workspace/source/internal/datacoord/task_scheduler.go:423\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.7.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) dim not found in params\nError types: (1) *withstack.withStack (2) *errutil.leafError"}
{"level":"WARN","time":"2025/03/26 06:17:03.660 +00:00","caller":"rootcoord/meta_table.go:577","message":"not found collection","collectionID":453584804738020460}
{"level":"WARN","time":"2025/03/26 06:17:06.660 +00:00","caller":"rootcoord/meta_table.go:577","message":"not found collection","collectionID":453584804738020460}
{"level":"WARN","time":"2025/03/26 06:18:04.980 +00:00","caller":"datacoord/metrics_info.go:99","message":"channel not found in meta cache","channel":"by-dev-rootcoord-dml_3_453584804738020460v0"}
{"level":"WARN","time":"2025/03/26 06:18:06.660 +00:00","caller":"rootcoord/meta_table.go:577","message":"not found collection","collectionID":453584804738020460}

so i guess some redundant pieces are remaining. may be line number in loglines can highlight where is the issue?

[xiaofan-luan]

did you restart all the node? especially root and datacoord

[artesby]

i restarted all components except querynodes and finally warnings are gone. thank you so much for you help

[xiaofan-luan]

i restarted all components except querynodes and finally warnings are gone. thank you so much for you help

Great!
Enjoy the new 2.5 release! Let me know if you found your search running faster~