Server seems to crash when opening mp4(or large) files #265

New Issue

michael · 2026-01-19T18:29:40Z

michael commented

2026-01-19 18:29:40 +00:00

Originally created by @kiska3 on GitHub.

Crash report: https://pastebin.com/YxMCnXQd

The server seems to crash when opening a mp4 file from local storage, unsure about cause.

Originally created by @kiska3 on GitHub. Crash report: https://pastebin.com/YxMCnXQd The server seems to crash when opening a mp4 file from local storage, unsure about cause.

michael closed this issue

2026-01-19 18:29:40 +00:00

michael commented

2026-01-19 18:29:40 +00:00

@JustAnotherArchivist commented on GitHub:

Oh, I see. Sorry about the noise; I was just going by the documentation for ioutil.ReadAll.

@JustAnotherArchivist commented on GitHub: Oh, I see. Sorry about the noise; I was just going by the [documentation for ioutil.ReadAll](https://golang.org/pkg/io/ioutil/#ReadAll).

michael commented

2026-01-19 18:29:40 +00:00

@JustAnotherArchivist commented on GitHub:

The GET handler reads the entire file to memory and then doesn't do anything with the read data:

cd602b9256/server/handlers.go (L905)

That code was added in af80d889 to support Range requests. I'm not familiar with Go or the meaning behind that bit of the code, but there has to be a better way than reading the entire file to memory and discarding it.

@JustAnotherArchivist commented on GitHub: The GET handler reads the entire file to memory and then doesn't do anything with the read data: https://github.com/dutchcoders/transfer.sh/blob/cd602b9256fb77e705da3cf6e09323e2eab233f6/server/handlers.go#L905 That code was added in af80d889 to support Range requests. I'm not familiar with Go or the meaning behind that bit of the code, but there has to be a better way than reading the entire file to memory and discarding it.

michael commented

2026-01-19 18:29:40 +00:00

@kiska3 commented on GitHub:

This is the command I am using: while true; do date; ./transfersh --provider=local --listener :8080 --temp-path=/transfersh/ --basedir=/transfersh/; sleep 10; done

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 39G 11G 29G 27% /

Using ps aux | grep transfersh and free:
https://transfer.notkiska.pw/IvPnt/ouput.txt
http://transfer.notkiska.pw/i1rTs/ouput.txt
http://transfer.notkiska.pw/pjqBq/ouput.txt

Way I am doing to generate the output file: while true; do date; ps aux | grep transfersh; free; sleep 0.1; done > output

The server crashes using either cURL or in a browser. I would link the file, but I would like to not rip out my hair.

Server details:
1x Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
1GB RAM
40GB "SSD" space
The $5 lightsail tier

@kiska3 commented on GitHub: This is the command I am using: `while true; do date; ./transfersh --provider=local --listener :8080 --temp-path=/transfersh/ --basedir=/transfersh/; sleep 10; done` > Filesystem Size Used Avail Use% Mounted on > /dev/xvda1 39G 11G 29G 27% / Using ps aux | grep transfersh and free: https://transfer.notkiska.pw/IvPnt/ouput.txt http://transfer.notkiska.pw/i1rTs/ouput.txt http://transfer.notkiska.pw/pjqBq/ouput.txt Way I am doing to generate the output file: `while true; do date; ps aux | grep transfersh; free; sleep 0.1; done > output` The server crashes using either cURL or in a browser. I would link the file, but I would like to not rip out my hair. Server details: 1x Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz 1GB RAM 40GB "SSD" space The $5 lightsail tier

michael commented

2026-01-19 18:29:40 +00:00

@paolafrancesca commented on GitHub:

the code doesn't read the entire file to memory, if you check the ioutil.ReadAll has io.TeeReader as argument: https://github.com/golang/go/blob/master/src/io/io.go#L520-L524

I basically dump from the io.Reader return by storage.Get to a io.ReadSeeker that's passed to http.ServeContent

also, from the stack trace it seems you are not using local storage but s3 (with minio)
the file streamed from the storage (either local, s3 or gdrive) is written to temp path, can you check how much space do you have here?

also, large files doesn't really give any insight: how big is the file that produces the panic?
the panic is occurring on every request or only on some for the same file?
mp4 only files on the preview page are affected or also other kind of file?
what if you download directly the file instead of opening it in the preview page? does the panic occur anyway?

@paolafrancesca commented on GitHub: the code doesn't read the entire file to memory, if you check the `ioutil.ReadAll` has `io.TeeReader` as argument: https://github.com/golang/go/blob/master/src/io/io.go#L520-L524 I basically dump from the `io.Reader` return by `storage.Get` to a `io.ReadSeeker` that's passed to `http.ServeContent` also, from the stack trace it seems you are not using local storage but s3 (with minio) the file streamed from the storage (either local, s3 or gdrive) is written to temp path, can you check how much space do you have here? also, large files doesn't really give any insight: how big is the file that produces the panic? the panic is occurring on every request or only on some for the same file? mp4 only files on the preview page are affected or also other kind of file? what if you download directly the file instead of opening it in the preview page? does the panic occur anyway?

michael commented

2026-01-19 18:29:40 +00:00

@paolafrancesca commented on GitHub:

actually is ioutil.ReadAll that loads everything in memory, I will push a branch with the fix as soon as I finish testing it

@paolafrancesca commented on GitHub: actually is `ioutil.ReadAll` that loads everything in memory, I will push a branch with the fix as soon as I finish testing it

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dutchcoders/transfer.sh#265