Writeback on a nvme drive

[FallingSnow]

I have a 16TB HDD and a 2TB NVME. I'm hoping to use rapiddisk to accelerate a combination of the 2.

From what I can tell rapiddisk is mainly focused on being used with a ram disk. However I'd like to use my nvme drive as a write cache and perhaps a read cache too (50% of the drive for each).

Is this something that is possible with rapiddisk? Or does rapiddisk only work with ramdisks?

I would use mdadm's write journal but seems to only work with raid 4/5/6 but I only want to use raid 1 across an nvmeof network.

[pkoutoupis]

@FallingSnow Thank you for your interest but unfortunately, RapidDisk was not designed for this. Yes, it only works with RAM drives. What I would recommend is looking into dm-writecache. I wrote an article about this some time back: https://www.admin-magazine.com/HPC/Articles/Linux-Writecache. At least for the 50% of the write cache you wish to set up. As for a read cache, off the top of my head, I do not know if you can configure dm-cache with a write-through or write around caching policy but it is definitely something you can look into.

[pkoutoupis]

That really sucks about the instability. That is a shame though. Hopefully you will be able to update the distro in the near future to get more stable versions of these packages and modules.

Anyway, to answer your questions:

RapidDisk originally started out as a much more flexible RAM drive module. It was modeled after the md module of BSD and ramdiskadm flexibility of Solaris. The caching piece came a bit later because:

• At the time, the block layer (in the kernel) did not have a good caching environment (for raw block devices) and it served more like a very temporary buffer cache. That is, unless you used a local file system. The caching was a bit more well instrumented in the VFS. Although, in both cases, under memory pressure, cache memory was reclaimed. RapidDisk-Cache tries to maintain its own cache environment with its own cache policy, etc. It is not subject to the rules of memory reclamation in the kernel.
• ALSO, RapidDisk/RapidDisk-Cache was originally designed to export these block devices (RAM-only or cached block device) across a SAN (i.e. iSCSI and now NVMeoF). That made the need to have a more consistent and controlled cache environment more necessary.

My opinion: bcache vs. dm-writecache? Hmmm. This is a tough one. I have very little experience with bcache. My bias will direct you to dm-writecache but if it is unstable in your environment, then that may not be the best choice for you. I do believe that bcache formats your volumes similar to a file system, so it may be more difficult to migrate or detach the cache volume and plug the backend target into something else (with LVM). This claim may require a bit of research on your part.

[FallingSnow]

Interesting. Thank you for your thought-out response. I hope to get a new lts version in the next couple of months and will be updating once that happens. Based on your recommendation I hope to give dm-writecache a good chance once that happens.

NVMeoF is becoming ever more common and is something I'm interested in so I will have to keep RapidDisk in the back of my mind for the future.

...so it may be more difficult to migrate or detach the cache volume and plug the backend target into something else (with LVM)

I'll have to do some testing but based the kernel docs it seems straightforward, however I'm not sure how it's done on LVM.

My only gripe with bcache is a lack of options similar to dm-cache's mq policy write_promote_adjustment and read_promote_adjustment to determine cache priority. Mostly because I'd like to have a nvme write cache mostly just allow fast random and sequential writes, returning cached writes only. Kind of like a writeback + read cache w/ no read promotion. Then another layer with a writearound read cache. This way I can control the cache allocations and prevent a bunch of sequential writes pushing out the cached random reads.

Thank you again.

[pkoutoupis]

You are welcome.

[nazar-pc]

@FallingSnow something is odd with your Alpine Linux installation, linux-lts package is at 5.15 already in latest stable release 3.16 and linux-edge is at 6.0.5 right now. Just upgrading your installation should result in much newer kernel.

[FallingSnow]

Oops. I think I miss typed my version number. I'm on 5.15 linux-lts and I can't use edge because of other requirements. Unfortunately alpine 3.16 has a 2+ year old version of lvm2.