FileStream and general performance (esp network)

[rmsimpson]

I've spent a few days trying to figure out how to speed up FileStream a bit and wondered if anyone else has run into these issue(s) I've been having ...

On 10gbit ethernet, I can't get FileStream to go above about 3.5gbit/sec on its own. No buffer changes, no open flags, no combination will get FileStream up high enough to saturate a 10gbit connection.

On powershell for example, if you run Get-FileHash on a network file over 10gbit, you won't go over 3.5. On local SSD it's fairly speedy, but over a network it's a dog.

My conclusion so far has been that the FileStream implementation, buffered or unbuffered, just isn't keeping read requests queued up enough to saturate the wire. Below it took over 1 minute to SHA256 hash a 21gb file over the network. It spent only 13 seconds in the hash algorithm, the rest was FileStream. For comparison, it normally takes roughly 25 seconds to physically copy the file from the network to local.

The code's fairly simple, I've tried dozens of iterations, from async to non-async, Memory<T> Span<T> you name it. Results vary only slightly.

private static void HashStream(Stream stream)
{
	Stopwatch sw = new();
	Stopwatch sh = new();
	
	var block = new byte[0x40000];
	using var hasher = SHA256.Create();

	int readlen = 1;
	while (readlen > 0)
	{
		sw.Start();
		readlen = stream.Read(block, 0, block.Length);
		sw.Stop();

		sh.Start();
		hasher.TransformBlock(block, 0, readlen, null, 0);
		sh.Stop();
	}
	hasher.TransformFinalBlock(block, 0, 0);
	var hash = hasher.Hash;

	Console.WriteLine($"{stream.GetType().FullName} Took {sw.Elapsed} to read");
	Console.WriteLine($"{stream.GetType().FullName} Took {sh.Elapsed} to hash {Convert.ToHexString(hash ?? Array.Empty<byte>())}");
	Console.WriteLine($"{stream.GetType().FullName} total time was {sw.Elapsed + sh.Elapsed}");
}

[huoyaoyuan]

What's the speed accessing the file share with other methods? For example, copying with PowerShell, explorer and some third-party file manager?
The difference between local file and file share is hidden by the OS. You can also map the file share to a volume letter and it shows no difference to application.

[rmsimpson]

File.Copy executes full speed, 10gbit, as expected so does Windows Explorer. Copying file with powershell uses the same Copy logic as Explorer and File.Copy so it runs full speed. It's only when you use FileStream that it caps at about 3.5gbit.

[huoyaoyuan]

@dotnet/ncl for any idea

[tmds]

readlen = stream.Read(block, 0, block.Length);

Sequential reading costs network round trips.

You can probably increase the speed by using larger buffers, and performing reads in parallel.

[huoyaoyuan]

0.25MB isn't that small. Can the hasher cause delay between packets?

[tmds]

10 gbps is roughly 1000 MBps. Thats rougly 4000 0.25MB round trips per second.

To have more date in flight, you can use larger buffers and/or perform reads in parallel.

[tmds]

Can the hasher cause delay between packets?

It adds delay between requests like the round trip. The round trip is expected to be larger source of delay.

[danmoseley]

How fast is FileStream when the source and destination are local?

[rmsimpson]

When they are local, it's very fast. Takes roughly 15 seconds to perform the same filehash on local SSD.

[wfurt]

Well you talk about FileStream but your next picture shows ethernet @rmsimpson. How is that related? If you do transfer from file via network you should post full code. You an also experiment with MemoryStream to se where the bottlenecks are.

And as @tmds suggested larger transfers typically yield better results. You can increase send/receive socket buffers and your should disable Nagle's e.g NoDelay=true

[rmsimpson]

So, here's another hash run with a 1 gigabyte buffer. You can see the spikes up and down when the system has to wait for the buffer to get refilled. This is with Asynchronous, SequentialScan, and FileFlagNoBuffering (which actually helps a lot).

@wfurt The only thing I left out of the code earlier was the stream open, so here you go:

const FileOptions FileFlagNoBuffering = (FileOptions)0x20000000;
using var stream = new FileStream(sourceFile, new FileStreamOptions()
{
	Access = FileAccess.Read,
	Mode = FileMode.Open,
	Share = FileShare.Read,
	Options = FileOptions.Asynchronous | FileFlagNoBuffering | FileOptions.SequentialScan,
	BufferSize = 0x40000000,
});
HashStream(stream);

And yes, I've tried every Options flag.
And yes, I'm talking about how FileStream performs over a network connection. Nagle plays no part in this, I'm not opening a socket, I'm reading a file over an SMB 3.0 share on a 10gbit network.

[rmsimpson]

I spent yesterday writing a POC to use buckets instead. This is the same run on the same file, using 32x512kb buckets instead of a linear buffer. The results were staggering and went far beyond my expectations, and even sped up local SSD hashing with only a total of 16mb of buffer space:

I'm not posting the code (yet, it's still really green), but I basically just "wrapped" filestream, and put my bucketreader on top of it.

using var stream = new FileStream(sourceFile, new FileStreamOptions()
{
	Access = FileAccess.Read,
	Mode = FileMode.Open,
	Share = FileShare.Read,
	Options = FileOptions.Asynchronous | FileFlagNoBuffering | FileOptions.SequentialScan,
	BufferSize = 0,
});
using var bucket = new BucketFileStreamReader(stream, 32, 524288);
HashStream(bucket);

[rmsimpson]

Local SSD hashing the same file, performance gains ...

BucketStreamReader w/ 32x512kb buckets:

ConsoleApp1.BucketFileStreamReader Took 00:00:01.4244530 to read
ConsoleApp1.BucketFileStreamReader Took 00:00:08.9720030 to hash DCA81DB01791454F7A59CB0E316E5EFB919EB9ACDE8851654B46796A5142CDC0
ConsoleApp1.BucketFileStreamReader total time was 00:00:10.3964560
83697 reads and 0 writes

FileStream w/ a single 16mb buffer:

System.IO.FileStream Took 00:00:07.5555781 to read
System.IO.FileStream Took 00:00:08.5522134 to hash DCA81DB01791454F7A59CB0E316E5EFB919EB9ACDE8851654B46796A5142CDC0
System.IO.FileStream total time was 00:00:16.1077915

So, the bucketstream reader cut about 6 seconds off the read portion of the file hashing