S3 manager reading entire file into ram

[sunilke]

Hello , was going through one issue of s3 manager to upload a file it reads the entire file is read into RAM before it's being sent:
aws/aws-sdk-go#2228

Is anything planned for future to move away from s3 manager because of this or its already solved in sftpgo using some way/hack?

[drakkan]

Hi,

I'm moving to aws sdk v2, however I think you are wrong, the memory usage is upload_concurrency * upload_part_size and the same for downloads. You can easily test yourself. This behaviour will remain the same also with aws sdk v2.

Before asking for a question or reporting an issue please do some tests, thanks.

P.S. are you really using SFTPGo?

[drakkan]

Sure , thanks.

Also:

P.S. are you really using SFTPGo?

Currently its more of a exploration

I might be wrong, but based on your questions I get the impression that you want to understand the SFTPGo internals to do something similar. Of course you are free to do this but:

1. I have limited free time and want to help SFTPGo users. If I don't answer your next question, you know why
2. SFTPGo license is AGPLv3: you can study it but you cannot copy the SFTPGo code inside a proprietary application

thanks for understanding

[sunilke]

No not at all , why would I work on something that's already there. We want to use sftpgo itself and that's why I am exploring all the things it offers.

The main reason I asked you above question is once anyone starts using sftpgo and there comes situation were uploads/downloads start failing so someone needs to have basic understanding to locate the error, though we can everytime ask you in discussion/issue section whatever is appropriate but in some cases we want quick resolutions before enough harm is done.

If I get the understanding of basic flows then I might be able to locate any critical errors faced while using it. I saw multiple closed issues/discussions around this temp directory so thought of clearing it out 100% as someone using sftpgo will most probably have ask of uploading/downloading big files and if any problems arise in future while doing such operation then having good understanding will certainly help to explain the reason/s.

Sorry if you felt like I am trying to do something similar because I am not sure how to prove it to you that I never have thought of doing so and even not capable for that

[sunilke]

For now I just use sftpgo in my local environment using s3 as underline sftp's storage and every uploads/downloads works fine . I just do uploads/downloads of small size files (max 1 gb) but in real life scenario files can be very large so I was keen that I know through and through of s3fs uploads/downloads.

But yes as you said I must do enough testing on my side before raising questions, I will totally follow it

[drakkan]

For now I just use sftpgo in my local environment using s3 as underline sftp's storage and every uploads/downloads works fine . I just do uploads/downloads of small size files (max 1 gb) but in real life scenario files can be very large so I was keen that I know through and through of s3fs uploads/downloads.

But yes as you said I must do enough testing on my side before raising questions, I will totally follow it

Hi,

if you observe the ram usage you can easily see that SFTPGo will not use 1GB ram if you upload a 1GB file. As discussed in the past the only things to keep in mind is that if you need to upload a 10GB file you need 10GB of local space for the (temporary) unliked file.

I just rewrote the s3fs implementation to use AWS SDK v2. It worked fine in my tests and the new SDK is much more idiomatic than v1 but this implentation has not yet real world testing. If you want to help the project you could test the development version and report any issue you find, thanks

[sunilke]

Yes, I will surely try to do it and will update if done anytime .
Yet again I have never thought of developing something similar because I understand your and other devs hardwork who have contributed to sftpgo and this project is much advance that can cater everyone's need . Also I can give you one assurance that in future you will be seeing many good(and I hope big too) contributions on this project from my side.

Thanks a lot for helping me with each of my (even silly sometimes) questions and issues. Hope this project reaches to yet more large masses.