dutchcoders/transfer.sh
1
0
Fork 0
mirror of https://github.com/dutchcoders/transfer.sh.git synced 2026-02-03 06:03:25 +00:00
Code Issues 48 Actions Packages Projects Releases Wiki Activity

How can I resume downloads? #106

New Issue
Open
opened 2026-01-19 18:28:59 +00:00 by michael · 16 comments
michael commented 2026-01-19 18:28:59 +00:00
Owner
Copy Link

Originally created by @mptnan on GitHub.

I'm downloading from transfer.sh url like wget -c "https://transfer.sh/..." but in the middle way an error occurs stating that "...バイトで読み込みエラーが発生しました (Operation timed out)。 再試行しています。".
Then it restarts downloading but from the scratch (progress bar starts from 0%). I read this https://github.com/dutchcoders/transfer.sh/issues/289 but didn't get the solution.

Originally created by @mptnan on GitHub. I'm downloading from transfer.sh url like wget -c "https://transfer.sh/..." but in the middle way an error occurs stating that "...バイトで読み込みエラーが発生しました (Operation timed out)。 再試行しています。". Then it restarts downloading but from the scratch (progress bar starts from 0%). I read this https://github.com/dutchcoders/transfer.sh/issues/289 but didn't get the solution.
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@mptnan commented on GitHub:

Sorry but I'm always putting "wget -c url" and what's wrong with that

@mptnan commented on GitHub: Sorry but I'm always putting "wget -c url" and what's wrong with that
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

hello @niss88

after some tests I can confirm that the code handles download resume on video and audio file (we could change for every file type, but that's no related to your problem)

failure on download resume at https://transfer.sh could be due to a proxy that doesn't forward the range header (it was indeed the same on my private instance): @stefanbenten , can you validate this assumption?
do you have a proxy in front of the service?

in case of nginx I had to add the proxy_pass_request_headers on; directive, btw

@paolafrancesca commented on GitHub: hello @niss88 after some tests I can confirm that the code handles download resume on video and audio file (we could change for every file type, but that's no related to your problem) failure on download resume at https://transfer.sh could be due to a proxy that doesn't forward the range header (it was indeed the same on my private instance): @stefanbenten , can you validate this assumption? do you have a proxy in front of the service? in case of nginx I had to add the `proxy_pass_request_headers on;` directive, btw
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

from wget documentation

-c
--continue
    Continue getting a partially-downloaded file.  This is useful when you want to finish up a download started by a previous instance of Wget, or by another program.  For instance:

            wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z

    If there is a file named ls-lR.Z in the current directory, Wget will assume that it is the first portion of the remote file, and will ask the server to continue the retrieval from an
    offset equal to the length of the local file.

    Note that you don't need to specify this option if you just want the current invocation of Wget to retry downloading a file should the connection be lost midway through.  This is the
    default behavior.  -c only affects resumption of downloads started prior to this invocation of Wget, and whose local files are still sitting around.

    Without -c, the previous example would just download the remote file to ls-lR.Z.1, leaving the truncated ls-lR.Z file alone.
@paolafrancesca commented on GitHub: from wget documentation ``` -c --continue Continue getting a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of Wget, or by another program. For instance: wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z If there is a file named ls-lR.Z in the current directory, Wget will assume that it is the first portion of the remote file, and will ask the server to continue the retrieval from an offset equal to the length of the local file. Note that you don't need to specify this option if you just want the current invocation of Wget to retry downloading a file should the connection be lost midway through. This is the default behavior. -c only affects resumption of downloads started prior to this invocation of Wget, and whose local files are still sitting around. Without -c, the previous example would just download the remote file to ls-lR.Z.1, leaving the truncated ls-lR.Z file alone. ```
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@mptnan commented on GitHub:

My downloading file was a video... but I'll wait either way.

For testing I uploaded mp4 file for example https://transfer.sh/KnDl1c/out3.mp4 (~1 MB), tested like below and --continue (-c) didn't work for mp4 file

*****.. [13:56:54] tmp% ls                                         
*****.. [13:57:40] tmp% wget -c https://transfer.sh/KnDl1c/out3.mp4
--2022-06-03 13:57:55--  https://transfer.sh/KnDl1c/out3.mp4
transfer.sh (transfer.sh) をDNSに問いあわせています... 144.76.136.153, 2a01:4f8:200:1097::2
transfer.sh (transfer.sh)|144.76.136.153|:443 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 200 OK
長さ: 942941 (921K) [video/mp4]
`out3.mp4' に保存中

out3.mp4                                     87%[=============================================================================>            ] 807.41K   225KB/s   残り1s      ^C
*****.. [13:58:01] tmp% wget -c https://transfer.sh/KnDl1c/out3.mp4
--2022-06-03 13:58:07--  https://transfer.sh/KnDl1c/out3.mp4
transfer.sh (transfer.sh) をDNSに問いあわせています... 2a01:4f8:200:1097::2, 144.76.136.153
transfer.sh (transfer.sh)|2a01:4f8:200:1097::2|:443 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 200 OK
長さ: 942941 (921K) [video/mp4]
`out3.mp4' に保存中

out3.mp4                                     32%[===========================>                                                              ] 295.37K   177KB/s               ^C
*****.. [13:58:11] tmp%
@mptnan commented on GitHub: My downloading file was a video... but I'll wait either way. For testing I uploaded mp4 file for example https://transfer.sh/KnDl1c/out3.mp4 (~1 MB), tested like below and --continue (-c) didn't work for mp4 file ``` *****.. [13:56:54] tmp% ls *****.. [13:57:40] tmp% wget -c https://transfer.sh/KnDl1c/out3.mp4 --2022-06-03 13:57:55-- https://transfer.sh/KnDl1c/out3.mp4 transfer.sh (transfer.sh) をDNSに問いあわせています... 144.76.136.153, 2a01:4f8:200:1097::2 transfer.sh (transfer.sh)|144.76.136.153|:443 に接続しています... 接続しました。 HTTP による接続要求を送信しました、応答を待っています... 200 OK 長さ: 942941 (921K) [video/mp4] `out3.mp4' に保存中 out3.mp4 87%[=============================================================================> ] 807.41K 225KB/s 残り1s ^C *****.. [13:58:01] tmp% wget -c https://transfer.sh/KnDl1c/out3.mp4 --2022-06-03 13:58:07-- https://transfer.sh/KnDl1c/out3.mp4 transfer.sh (transfer.sh) をDNSに問いあわせています... 2a01:4f8:200:1097::2, 144.76.136.153 transfer.sh (transfer.sh)|2a01:4f8:200:1097::2|:443 に接続しています... 接続しました。 HTTP による接続要求を送信しました、応答を待っています... 200 OK 長さ: 942941 (921K) [video/mp4] `out3.mp4' に保存中 out3.mp4 32%[===========================> ] 295.37K 177KB/s ^C *****.. [13:58:11] tmp% ```
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

I'm sorry @niss88, you are right
range donwload works only for video and audio file: https://github.com/dutchcoders/transfer.sh/blob/main/server/handlers.go#L1054

I will fix as soon as I have time

@paolafrancesca commented on GitHub: I'm sorry @niss88, you are right range donwload works only for video and audio file: https://github.com/dutchcoders/transfer.sh/blob/main/server/handlers.go#L1054 I will fix as soon as I have time
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

@stefanbenten
taking @niss88 upload (https://transfer.sh/KnDl1c/out3.mp4), seeking does not work on the video player, that should be related to the range header

@paolafrancesca commented on GitHub: @stefanbenten taking @niss88 upload (https://transfer.sh/KnDl1c/out3.mp4), seeking does not work on the video player, that should be related to the `range` header
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@ErrorNoInternet commented on GitHub:

This is also affecting parallel download tools (tools that connect to the server multiple times and download segments of the file in each thread). It makes a HEAD request to see if the server gives the Accept-Ranges: bytes header, and then sends a GET request containing the Range: bytes=0-N header to download the first N bytes.

@ErrorNoInternet commented on GitHub: This is also affecting parallel download tools (tools that connect to the server multiple times and download segments of the file in each thread). It makes a `HEAD` request to see if the server gives the `Accept-Ranges: bytes` header, and then sends a GET request containing the `Range: bytes=0-N` header to download the first N bytes.
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

filtering out range header to the transfer.sh service will also break seeking feature on the audio/video preview page

@paolafrancesca commented on GitHub: filtering out `range` header to the transfer.sh service will also break seeking feature on the audio/video preview page
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@stefanbenten commented on GitHub:

transfer.sh is currently running with nginx in front and the correct proxy pass directive added. That said, it has a timeout at 1800s per request, due to the fact that there have been HTTP slow attacks in the past.

@stefanbenten commented on GitHub: transfer.sh is currently running with nginx in front and the correct proxy pass directive added. That said, it has a timeout at 1800s per request, due to the fact that there have been HTTP slow attacks in the past.
michael commented 2026-01-19 18:28:59 +00:00
Author
Owner
Copy Link

@stefanbenten commented on GitHub:

I think i have an idea why it is not working correctly. Lemme debug the code (because the proxy forwards it correctly) and put up a PR.

@stefanbenten commented on GitHub: I think i have an idea why it is not working correctly. Lemme debug the code (because the proxy forwards it correctly) and put up a PR.
michael commented 2026-01-19 18:29:00 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

@ErrorNoInternet did you hit this issue with transfer.sh, intended as software, or on https://transfer.sh website?

can you tell me one of the parallel download tools that are affected?

I will check if everything is fine on the transfer.sh side.

For sure Range: bytes= requests are currently supported only for video and audio content type. We can widen support for every file.

@paolafrancesca commented on GitHub: @ErrorNoInternet did you hit this issue with `transfer.sh`, intended as software, or on https://transfer.sh website? can you tell me one of the parallel download tools that are affected? I will check if everything is fine on the `transfer.sh` side. For sure `Range: bytes=` requests are currently supported only for video and audio content type. We can widen support for every file.
michael commented 2026-01-19 18:29:00 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

HEAD requests do not return Accept-Ranges: bytes, I will fix that, as well of allowing partial download for every file type

this is related to the support on transfer.sh intended as software.

hosted instances like https://transfer.sh, might or might nor support this according to the presence of a proxy and its setup

@paolafrancesca commented on GitHub: `HEAD` requests do not return `Accept-Ranges: bytes`, I will fix that, as well of allowing partial download for every file type this is related to the support on `transfer.sh` intended as software. hosted instances like https://transfer.sh, might or might nor support this according to the presence of a proxy and its setup
michael commented 2026-01-19 18:29:00 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

indeed it's a rather tricky: we have a mutex on checkMetadata (35e794220b/server/handlers.go (L689-L690)) in order to track the MaxDownloads and update them

parallel range requests will be handled sequentially anyway because of the mutex.
I cannot see an easy way to get rid of the mutex: just skipping it in case of range requests will be prone to abuse (I do an HEAD and just make a range one with a full range but one byte)

keeping track of all the requested ranges to understand when a full donwload happened is a no way to go, since there are a lot of cases where range requests are not a parallel download (seeking in a multimedia content for example)

skipping the lock if the upload doesn't contain MaxDownloads is troublesome as well: since we'll know that after checking the metadata, but if it was the case we should have acquired the lock. I will see if I can move the lock before updating the Downloads counter (we have to re-check the metadata just before indeed because of race conditions).
I have to find a way where this is producing all the required goals:

  1. block the donwload as soon as the N+1th request is made with N equal to MaxDownloads
  2. update Downloads in metadata without race conditions
  3. if no MaxDownloads is set allow for parallel downloads
@paolafrancesca commented on GitHub: indeed it's a rather tricky: we have a mutex on `checkMetadata` (https://github.com/dutchcoders/transfer.sh/blob/35e794220b3c998194d7d52796e052ef00795c2f/server/handlers.go#L689-L690) in order to track the `MaxDownloads` and update them parallel range requests will be handled sequentially anyway because of the mutex. I cannot see an easy way to get rid of the mutex: just skipping it in case of range requests will be prone to abuse (I do an `HEAD` and just make a range one with a full range but one byte) keeping track of all the requested ranges to understand when a full donwload happened is a no way to go, since there are a lot of cases where range requests are not a parallel download (seeking in a multimedia content for example) skipping the lock if the upload doesn't contain `MaxDownloads` is troublesome as well: since we'll know that after checking the metadata, but if it was the case we should have acquired the lock. I will see if I can move the lock before updating the `Downloads` counter (we have to re-check the metadata just before indeed because of race conditions). I have to find a way where this is producing all the required goals: 1. block the donwload as soon as the N+1th request is made with N equal to `MaxDownloads` 2. update `Downloads` in metadata without race conditions 3. if no `MaxDownloads` is set allow for parallel downloads
michael commented 2026-01-19 18:29:00 +00:00
Author
Owner
Copy Link

@ErrorNoInternet commented on GitHub:

@aspacca

did you hit this issue with transfer.sh, intended as software, or on https://transfer.sh/ website?

The https://transfer.sh website

can you tell me one of the parallel download tools that are affected?

Some examples are Paralload (my own tool), Chrome (it has a multi-threaded download feature), chunked-downloader, and a few more small tools.
My ISP throttles download speeds (sometimes down to 20 KB/s) for foreign sites, so I rely on these tools to download files faster.

@ErrorNoInternet commented on GitHub: @aspacca > did you hit this issue with transfer.sh, intended as software, or on https://transfer.sh/ website? The https://transfer.sh website > can you tell me one of the parallel download tools that are affected? Some examples are [Paralload](https://github.com/ErrorNoInternet/Paralload) (my own tool), Chrome (it has a multi-threaded download feature), [chunked-downloader](https://github.com/vbrown608/chunked-downloader), and a few more small tools. My ISP throttles download speeds (sometimes down to 20 KB/s) for foreign sites, so I rely on these tools to download files faster.
michael commented 2026-01-19 18:29:00 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

@ErrorNoInternet here: https://github.com/dutchcoders/transfer.sh/pull/495

@paolafrancesca commented on GitHub: @ErrorNoInternet here: https://github.com/dutchcoders/transfer.sh/pull/495
michael commented 2026-01-19 18:29:00 +00:00
Author
Owner
Copy Link

@paolafrancesca commented on GitHub:

@niss88 allowing parallel range request showed to be more complicated than I initially thought, especially regarding performance impact.

I will reserve to go back to the issue once I'll have more time for it

@paolafrancesca commented on GitHub: @niss88 allowing parallel range request showed to be more complicated than I initially thought, especially regarding performance impact. I will reserve to go back to the issue once I'll have more time for it
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
docker
enhancement
hacktoberfest
help wanted
invalid
question
storage provider
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
michael
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dutchcoders/transfer.sh#106
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?