Intermittent "File not found" errors on ZFS dataset during AI training

[qiuwei]

I created a static dataset on a ZFS pool (zpool) comprised of three NVMe SSDs (no raid). The dataset contains ~6000 files (each ~300 MiB) used for AI training workloads.

During training, processes intermittently fail with "File not found" errors on randomly accessed files. This occurs in ~20% of training jobs (1 in 5). After the error:

• The reported file is confirmed to exist and can be read immediately afterward.
• zpool status shows no errors, checksum errors, or pool degradation.

The issue is not consistently reproducible but severely impacts workflow reliability.

Environment
ZFS Version:
zfs-2.3.1-1
zfs-kmod-2.3.1-1

OS: [6.14.8-zabbly+ #ubuntu22.04]

Hardware: NVMe SSDs (Model: DAPUSTOR DPHV510),
Pool Configuration:

NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bypool     41.9T  1.62T  40.3T        -         -     0%     3%  1.00x    ONLINE  -
  sda      14.0T   511G  13.5T        -         -     0%  3.57%      -    ONLINE
  sdb      14.0T   511G  13.5T        -         -     0%  3.57%      -    ONLINE
  nvme0n1  14.0T   633G  13.4T        -         -     0%  4.42%      -    ONLINE

any suggestion on how I can diagnose the issue?