[Bug]: Large file (>8GB) downloads failing

[pjgreer]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).

Bug description

I cannot find it in the documentation, but is there a maximum download file size for SFTPgo? Right now I am trying to share some very large files >10GB and it is immediately at 7284MB. Is this related to the VM size, or is it a function of the SFTPgo server?

Steps to reproduce

1. upload a large >8GB file to the GCP subsystem.
2. using the CLI scp or sftp login and path.
3. wait until 7284MB and it errors out with a read remote: Failure. and then a close remote: Failure

BTW, I have over 100GB free on my local drive.

Expected behavior

The file downloads successfully.

SFTPGo version

2.6.2

Data provider

GCP

Installation method

Community RPM package

Configuration

Using all default configs

Relevant log output

scp: read remote "/XXXXX/202406_XXXXX_XX/158004_GTFiles.zip" : Failure
scp: close remote: Failure

What are you using SFTPGo for?

Small business (3-person firm with file exchange?)

Additional info

No response

[drakkan]

Hello,

many users use SFTPGo to transfer large files, so it is unlikely to be a bug. I will convert it into a discussion. I suggest to post the relevant SFTPGo logs, maybe someone will have time/motivation to look at them and help you. Thanks

[pjgreer]

I figured it out. it was an issue with the default local system drive for my VM being too small. I was not aware that it was copying the file from GCP to the local system and then sending via FTP.

So could there be a minimum server requirements added to the docs? Specifically mentioning that the local system disk acts as a cache for the files from GCS and should be larger than the largest file you will be transferring.

[harshal-vaidya-rozetta]

@pjgreer - Could you find out what is the exact path where sftpgo writes the file to on the VM before it makes it available for download? We found out that it is not /var/lib/sftpgo.

[drakkan]

I figured it out. it was an issue with the default local system drive for my VM being too small. I was not aware that it was copying the file from GCP to the local system and then sending via FTP.

So could there be a minimum server requirements added to the docs? Specifically mentioning that the local system disk acts as a cache for the files from GCS and should be larger than the largest file you will be transferring.

For those interested, our Enterprise edition can be configured to avoid using unlinked disk files during transfers to cloud storage backends.