Documentation
¶
Overview ¶
Package httpx provides a shared HTTP client and a retry helper used by every scraper. Keeping this in one place means retry/backoff, timeout, and connection-pool tuning live in exactly one file.
Index ¶
- Constants
- Variables
- func BrowserHeaders(ua string) map[string]string
- func DecodeJSON(r io.Reader, v any) error
- func DecodeJSONN(r io.Reader, v any, maxBytes int64) error
- func Do(ctx context.Context, client *http.Client, r Request) (*http.Response, error)
- func DoJSON(ctx context.Context, client *http.Client, r Request, v any) error
- func DoWithStatus(ctx context.Context, client *http.Client, r Request) (*http.Response, error)
- func NewBrowserTLSClient(timeout time.Duration) *http.Client
- func NewClient(timeout time.Duration) *http.Client
- func NewLegacyTLSClient(timeout time.Duration) *http.Client
- func NewRetryClient(timeout time.Duration) *http.Client
- func ReadBody(body io.ReadCloser) ([]byte, error)
- func ReadBodyN(body io.ReadCloser, maxBytes int64) ([]byte, error)
- func RequestID(ctx context.Context) uint64
- func ResolveUA(ua string) string
- func SetDefaultUA(ua string)
- func WithRequestID(ctx context.Context) context.Context
- func XHRHeaders(ua, origin string) map[string]string
- type MalformedJSONError
- type Request
- type StatusError
Constants ¶
const MaxPageBytes = 10 * 1024 * 1024
MaxPageBytes caps ReadBody response reads to prevent an oversized or malicious response from exhausting memory.
Variables ¶
var ( UserAgentFirefox = builtinFirefox UserAgentChrome = builtinChrome )
UserAgentFirefox and UserAgentChrome are the active UA strings used by scrapers. Call SetDefaultUA at startup to override both with a config value.
Functions ¶
func BrowserHeaders ¶
BrowserHeaders returns headers that mimic a real browser navigation request, including Sec-Fetch-* headers that WAFs like Wordfence check. Pass a UA string, or empty to use the active default.
func DecodeJSON ¶
DecodeJSON JSON-decodes from r into v, reading at most MaxPageBytes.
func DecodeJSONN ¶
DecodeJSONN JSON-decodes from r into v, reading at most maxBytes.
Reads one byte past the limit so an oversized body is reported as such rather than surfacing as a confusing "unexpected EOF" from the decoder. A body that decodes successfully within the limit is fine even if bytes remain unread, so the size is only reported when decoding actually failed.
func Do ¶
Do performs the request with exponential backoff: it retries network errors, 429, and 5xx up to MaxAttempts times, sleeping (attempt * 2s) between tries. Non-retryable 4xx responses fail fast with a *StatusError — the caller does not have to guard against decoding an error page as a successful body.
func DoJSON ¶ added in v1.31.0
DoJSON performs the request like Do and decodes the response body into v, treating a body that is not JSON at all as a retryable failure.
An edge, WAF or load balancer that hiccups answers a JSON endpoint with an HTML error page under HTTP 200. Do sees a success status and hands the response back, the decode fails with `invalid character '<'`, and because a scraper's fetch error aborts the whole pagination walk, one such blip costs every remaining page. Those bodies are a transport failure wearing a parse error's clothes, so they share Do's attempt budget and backoff ladder.
Only a malformed body is retried. Well-formed JSON that does not fit v is the server's real answer and fails immediately — retrying it would just ask the same question three times. The final error names the status and quotes the start of what arrived, since the point of failure is usually obvious from the first few bytes and nothing else records them.
func DoWithStatus ¶
DoWithStatus is like Do but passes any HTTP status (including 4xx and 5xx) through to the caller without classifying — useful for endpoints that legitimately return non-2xx with a meaningful body (e.g. SexMex's CMS returns HTTP 500 + valid HTML on model pages). Network errors are still retried with the same backoff as Do, but 429/5xx are NOT retried — the caller asked for the raw response and presumably wants to act on it. Default for `Do` (4xx fail-fast, 429/5xx retried) is the safer choice for everything else.
func NewBrowserTLSClient ¶ added in v1.30.0
NewBrowserTLSClient returns a client that presents a browser's TLS fingerprint instead of Go's.
Reach for it only after a scraper has been shown to fail without it — the signature is a WAF answering an identical 403 to every request including the site's own homepage, while a browser on the same machine loads it. It is the mirror image of NewLegacyTLSClient: that one widens what Go will accept from an old server, this one changes what Go presents to a picky one. Both verify certificates.
The connection pool is separate from NewClient's, so using it for one site does not change how every other site is reached.
func NewLegacyTLSClient ¶
NewLegacyTLSClient returns a client for hosts that cannot complete a handshake with Go's default cipher list. Certificates are still verified — this widens the accepted key exchange, it does not disable checks. The negotiated suites lack forward secrecy, so use it only where a scraper has been shown to fail otherwise, and only for reading public metadata.
func NewRetryClient ¶ added in v1.30.0
NewRetryClient returns an http.Client that retries the way Do does — network errors, 429 and 5xx, with the same jittered backoff — for libraries that take an *http.Client and drive the request themselves.
timeout bounds each attempt rather than the sequence, so a retried request gets the full budget every time. Non-retryable responses, 4xx included, come back untouched with their body intact.
func ReadBody ¶
func ReadBody(body io.ReadCloser) ([]byte, error)
ReadBody reads an HTTP response body up to MaxPageBytes. Use this instead of io.ReadAll(resp.Body) in scrapers to bound memory usage.
func ReadBodyN ¶
func ReadBodyN(body io.ReadCloser, maxBytes int64) ([]byte, error)
ReadBodyN reads an HTTP response body up to maxBytes.
func ResolveUA ¶
ResolveUA maps a shorthand to a full UA string. "firefox" and "chrome" return the built-in strings; anything else is returned as-is.
func SetDefaultUA ¶
func SetDefaultUA(ua string)
SetDefaultUA overrides both exported UA variables. Accepts "firefox", "chrome", or a full custom string. Empty is a no-op (keeps built-ins). Call once at startup before any scrapers run.
func WithRequestID ¶
WithRequestID attaches an auto-incrementing request ID to the context. The ID appears in level-2 debug logs for correlation.
func XHRHeaders ¶ added in v1.31.0
XHRHeaders returns the headers a browser actually sends on a `fetch()` call to a same-origin API, for scrapers calling a JSON endpoint rather than loading a page. Pass a UA string (empty for the active default) and the site's origin (empty to omit it).
This exists because BrowserHeaders is wrong for an API call, and not harmlessly so. Its `Sec-Fetch-Dest: document` / `Sec-Fetch-Mode: navigate` / `Sec-Fetch-User: ?1` describe a top-level navigation, a thing no browser has ever expressed as a JSON POST — and a WAF that reads those headers can tell. MyDirtyHobby's answers such a request with a JavaScript cookie challenge under HTTP 200 — invisible as a status code, and it aborts a scrape mid-catalogue. Measured there over 25-request walks: the navigation values were challenged from the 18th request on, while these ran clean. Sending `empty`/`cors`/`same-origin` is not a disguise; it is the request describing itself accurately.
Types ¶
type MalformedJSONError ¶ added in v1.31.0
MalformedJSONError reports a response that arrived with a success status but whose body is not JSON — typically an edge or WAF error page served as HTTP 200 in place of an API answer. It carries the start of the body because nothing else records it: by the time the operator sees the warning the response is gone, and the first few bytes usually name the culprit.
func (*MalformedJSONError) Error ¶ added in v1.31.0
func (e *MalformedJSONError) Error() string
func (*MalformedJSONError) FailureKind ¶ added in v1.31.0
func (e *MalformedJSONError) FailureKind() scraper.FailureKind
FailureKind reports a body that arrived and could not be read as a parse failure. Retries have already been spent by the time this escapes DoJSON, so what reaches the operator is the site answering something FSS cannot use.
func (*MalformedJSONError) Unwrap ¶ added in v1.31.0
func (e *MalformedJSONError) Unwrap() error
Unwrap returns the decoder's own error so errors.Is/As still reach it.
type Request ¶
type Request struct {
Method string
URL string
Body []byte
Headers map[string]string
MaxAttempts int
// Note: there is deliberately no per-request body limit here. Do returns
// the response without reading the body, so it could not enforce one;
// bounding the read is the caller's job via ReadBody/ReadBodyN or
// DecodeJSON/DecodeJSONN.
BackoffSleep func(ctx context.Context, d time.Duration) error // nil uses default (real sleep with ctx)
}
Request describes a single HTTP call. Method defaults to GET (or POST if Body is non-nil). MaxAttempts defaults to 3.
type StatusError ¶
type StatusError struct {
StatusCode int
}
StatusError is returned when the server replies with a non-2xx status that we do not retry. The body has already been closed.
func (*StatusError) Error ¶
func (e *StatusError) Error() string
func (*StatusError) FailureKind ¶
func (e *StatusError) FailureKind() scraper.FailureKind
FailureKind reports that the page did not arrive, whatever the status was.
Deliberately including 404 and 410. It is tempting to call those "absent" and let them off the traversal-completeness hook, but a status code does not say whether the missing resource mattered: a 404 on an optional sub-listing the scraper was merely probing costs nothing, while a 404 on the detail page of a scene the listing already returned means a known scene went uncollected — and under `--full` the authoritative Save would then delete it. Only the call site knows which it is, so httpx always reports missing data and a scraper that knows better opts out explicitly with scraper.AbsentError.
Source Files