Partially ready sftp files from remote directory to reduce network traffic

[plewam]

Hi All,

First of a all thanks for creating this beautiful library.
I plan to periodically read generated log files from a remote directory. The logfiles itself are continuously appended. Once a threshold is reached the file is emptied and the appending continues (So the filename stays the same all the time)

In my current approach I do something like the below. I read the full file and check for differences/ new entries afterwards.
While this works I think this is rather bad from a performance/ network perspective. Ideally I would skip reading the 'already read' lines. What happens if I start reading with ReadAsync() at another index other than 0? Will the network traffic skipped for the offset bytes? Would that be a way to partially read the file?

await foreach (ISftpFile file in files)
{
        //--------------------------------------------------------------------
        // Start measuring time
        //--------------------------------------------------------------------
        Stopwatch watch = Stopwatch.StartNew();
        Logger.Log(LogLevel.Debug, "Found remote sftp file with name <{name}>", file.FullName);

        //--------------------------------------------------------------------
        // And open the file in read mode
        //--------------------------------------------------------------------
        using SftpFileStream s = client.OpenRead(file.FullName);
        stoppingToken.ThrowIfCancellationRequested();

        byte[] result = new byte[s.Length];

        //--------------------------------------------------------------------
        // This reads the whole file content...
        //--------------------------------------------------------------------
        Stopwatch readAsyncWatch = Stopwatch.StartNew();
        await s.ReadAsync(result, 0, (int)s.Length, stoppingToken).ConfigureAwait(false);
        Logger?.Log(LogLevel.Debug, "Successfully read <{bytes}> bytes (<{megaBytes}> mb) from file with name <{fileName}> in <{elapsedMilliseconds}> ms", result.LongLength, ByteSize.FromBytes(result.LongLength).MegaBytes, file.FullName, readAsyncWatch.ElapsedMilliseconds);
}

[Rob-Hague]

From reading the code, I think you could set SftpFileStream.Position after opening to get the desired behaviour, though I haven't tried it.

[plewam]

@Rob-Hague Thank you very much, that helped. I could've come up with this myself... Significantly reduces the network traffic.