Unix / Linux: Download Partial File over HTTP/HTTPS

Objective: Partially download a file or resume a broken download over HTTP/HTTPS on Unix / Linux.

In this article, we will look at two different scenarios. For the first scenario, we want to partially download a file. For the second scenario, we want to resume a a broken download. For both scenarios, we are going to use the curl utility.

Note that for this to work, the HTTP server must support partial downloads. Clients will use the HTTP Range header to specify ranges. HTTP servers advertise support of serving partial requests using the Accept-Ranges response header. When a range request is made and if it is valid, the HTTP servers will use the 206 Partial Content status code and a Content-Range header listing the range sent.

Partially Download a File

So, let’s say that we want to download a portion of a large file or the first 10kB of data. We can do this with the --range option of curl.

$ curl -O -r 0-9999 http://site.com/file.bin

1	$ curl -O -r 0-9999 http://site.com/file.bin

For the above command, we are telling curl to download the first 10kB bytes from a site and write to an output file called file.bin. Note that the range starts from 0.

If we need to download 10kB bytes of data after offset 10kB, or in other words download data between 10kB and 20kB, we will just need to modify the range parameter.

$ curl -O -r 10000-19999 http://site.com/file.bin

1	$ curl -O -r 10000-19999 http://site.com/file.bin

If we want to download the last 10kB of a file, then we can use the below syntax.

$ curl -O -r -10000 http://site.com/file.bin

1	$ curl -O -r -10000 http://site.com/file.bin

To download the whole file from offset 10kB, use the below syntax.

$ curl -O -r 10000- http://site.com/file.bin

1	$ curl -O -r 10000- http://site.com/file.bin

Resume Broken Download

To resume a broken file download, we can use the --continue-at option.

$ curl -O -C - http://site.com/file.bin

1	$ curl -O -C - http://site.com/file.bin

We use “-C -” to tell curl to automatically find out where/how to resume the transfer. You can also specify an offset to “-C”, but this is most often not necessary.

$ curl -O -C 10001 http://site.com/file.bin

1	$ curl -O -C 10001 http://site.com/file.bin

The above command will resume from offset 10001 and download the rest of the file.

Note that you can use this technique to split a large file into multiple sections and accelerate the download by running multiple curl processes with different ranges.

Mohamed Ibrahim

ibrahim = { interested_in(unix, linux, android, open_source, reverse_engineering); coding(c, shell, php, python, java, javascript, nodejs, react); plays_on(xbox, ps4); linux_desktop_user(true); }