How many ranges can you fit in one request
HTTP Range Requests are useful for serving HTML5 video and cloud geo formats, and use a header like this:
GET /example.pmtiles HTTP/2
Range: bytes=0-16383
The MDN docs on “HTTP Range Requests” suggest that you can use multiple ranges in a single request:
curl http://www.example.com -i -H "Range: bytes=0-50, 100-150"
This is specified in RFC 2616.
Single ranges work fine for videos, because their access pattern is linear: movies are designed to be played in one direction. More creative use cases for byte serving could take advantage of multiple ranges, by eliminating the latency overhead of multiple HTTP requests.
Bad News
The bad news is that almost all the static storage APIs, like Amazon S3, only support a single byte range per request.
This is true of HTTP endpoints e.g. bucket.s3.amazonaws.com/*
, as well as bindings to storage APIs like AWS SDK’s GetObjectCommand and GCP’s Storage API.
If you are cynical about these platforms you could infer a business reason behind this: most cloud storage charges a small amount per HTTP request. If you batched 100 ranges into 1, you have engineered yourself a 99% discount on transaction fees. This seems only somewhat likely, though, since transaction fees are low relative to bandwidth fees.
If you find a major S3-like storage system that supports MultiRanges, let me know!
Good News
The good news is that popular open source HTTP servers support HTTP MultiRanges out of the box, including Nginx, Caddy, and Apache.
But how many ranges can you fit in a single request?
Here’s a small program that asks for the first two bytes 2000 times in a single request:
package main
import (
"fmt"
"net/http"
"strings"
"slices"
)
func main() {
client := &http.Client{}
req, _ := http.NewRequest("GET", "http://example.com/bigfile", nil)
headerVal := "bytes=" + strings.Join(slices.Repeat([]string{"0-1"},2000),",")
req.Header.Set("Range", headerVal)
resp, _ := client.Do(req)
fmt.Println(resp.StatusCode, len(headerVal))
}
The server should return 206 Partial Content
; increasing the number of repeats will eventually return 400 Bad Request
or 431 Request Header Fields Too Large
.
The results match the documented max header size:
- for Apache and Nginx, the default limit of HTTP headers is 8 kilobytes total (including non
Range:
headers) - for Caddy, the default limit is 1 megabyte (128x!)
So the number of ranges in a single request depends on how much ASCII encoding space those numbers take up. You can fit a lot more ranges close to the beginning of a large file 0-100,200-300
than the end 1000000000-1000000100,2000000000-2000000100
.