mirror of
https://github.com/dutchcoders/transfer.sh.git
synced 2026-02-06 15:32:17 +00:00
Uploading file in chunks #291
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @cyberboysumanjay on GitHub.
I tried uploading a video of 1gb and I ran into a memory error in my heroku app. I did a quick research and found that this is because the transfer.sh service doesn't upload the file in multi-parts as the Google Drive API does so there is a high memory usage and app crashes. Is there any fix or a workaround for fixing this?
@paolafrancesca commented on GitHub:
@cyberboysumanjay the app use streaming io functions internally, so there should be no memory high memory allocation. I'm not sure about the part that uses the go drive API sdk and I will check further. if you can provide a stack trace of the crash it would be helpful
@paolafrancesca commented on GitHub:
@cyberboysumanjay I did some testing in a docker container with 10MB and 50MB of ram and tweaking the chunk size of the google sdk: I was able to reproduce an OOM issue with default chunk size (8MB) when the ram was limited to 10MB. With 50MB and default chunk size no issue, either with 10MB and minimum chunk size (256KB).
I also bumped the google sdk to the latest version, I will add as well a optional params to tweak the chunk size when using gdrive storage
@paolafrancesca commented on GitHub:
the google sdk already chunk the request: https://godoc.org/google.golang.org/api/googleapi#pkg-constants
we use https://godoc.org/google.golang.org/api/drive/v3#FilesCreateCall.Media
can you reference me where you found the problem is memory usage?