Compaction job failed

[rosh0450]

Compaction jobs are failing with this log

ts=2024-12-13T03:54:25.854586145Z caller=compact.go:807 level=error component=compactor msg="Failed to remove temp folder after failed compaction" dir=/data/compact/0@17241709254077376921-merge--1733961600000-1734004800000/01JEZ2EC0VNAQ02SBBB6ZJQZTG.tmp-for-creation err="unlinkat /data/compact/0@17241709254077376921-merge--1733961600000-1734004800000/01JEZ2EC0VNAQ02SBBB6ZJQZTG.tmp-for-creation: directory not empty"
ts=2024-12-13T03:54:25.893565792Z caller=bucket_compactor.go:271 level=error component=compactor user=tenant_okd groupKey=0@17241709254077376921-merge--1733961600000-1734004800000 job_type=merge minTime="2024-12-12 00:00:00.005 +0000 UTC" maxTime="2024-12-12 12:00:00 +0000 UTC" msg="compaction job failed" duration=1m30.199467284s duration_ms=90199 err="compact blocks 01JEWFT738266MH8RMN3BJMGBZ,01JEWQC50SEPZTM7QW21Z9NH93,01JEWY23G428K6AYC6QMEVJ6A9,01JEX4V3VBWJE7SRN0KKJNRRKM,01JEXBBNGZRWZZKN0AFXW1XAED,01JEXJ3QNEJX7WRC1K2NQKMS95: 2 errors: adding series: asyncBlockWriter doesn't run anymore: write chunks: sync /data/compact/0@17241709254077376921-merge--1733961600000-1734004800000/01JEZ2EC0VNAQ02SBBB6ZJQZTG.tmp-for-creation/chunks/000001: no space left on device; sync /data/compact/0@17241709254077376921-merge--1733961600000-1734004800000/01JEZ2EC0VNAQ02SBBB6ZJQZTG.tmp-for-creation/index: no space left on device" block_count=6
ts=2024-12-13T03:54:26.008861162Z caller=bucket_compactor.go:275 level=error component=compactor user=tenant_okd groupKey=0@17241709254077376921-merge--1733961600000-1734004800000 job_type=merge minTime="2024-12-12 00:00:00.005 +0000 UTC" maxTime="2024-12-12 12:00:00 +0000 UTC" msg="failed to remove compaction group work directory" path=/data/compact/0@17241709254077376921-merge--1733961600000-1734004800000 err="unlinkat /data/compact/0@17241709254077376921-merge--1733961600000-1734004800000/01JEZ2EC0VNAQ02SBBB6ZJQZTG.tmp-for-creation/.nfs000000006f30013e000000e5: device or resource busy"
ts=2024-12-13T03:54:36.407871304Z caller=bucket_compactor.go:930 level=info component=compactor user=tenant_okd msg="start sync of metas"
ts=2024-12-13T03:54:37.608707415Z caller=fetcher.go:500 level=info component=compactor user=tenant_okd component=block.MetaFetcher msg="successfully synchronized block metadata" duration=1.200819856s duration_ms=1200 cached=9 returned=9 partial=0
ts=2024-12-13T03:54:37.608726644Z caller=bucket_compactor.go:935 level=info component=compactor user=tenant_okd msg="start of GC"
ts=2024-12-13T03:54:37.611000915Z caller=bucket_compactor.go:976 level=warn component=compactor user=tenant_okd msg="failed deleting non-compaction job directories/files, some disk space usage might have leaked. Continuing" err="delete file/dir: delete file/dir: unlinkat /data/compact/0@17241709254077376921-merge--1733961600000-1734004800000/01JEZ2EC0VNAQ02SBBB6ZJQZTG.tmp-for-creation/.nfs000000006f30013e000000e5: device or resource busy" dir=/data/compact

One thing I can see here is it says "no space left on device" inside "/data" dir which is a PVC, but my pvc is empty.

Not sure what is happening.
I would appreciate someone's help.

Thanks