Documentation
¶
Overview ¶
Package scraper defines the interface for site scrapers and a global registry.
Each supported site implements StudioScraper and registers itself via Register in an init() function. Consumers look up scrapers with ForURL or ForID, then call StudioScraper.ListScenes to stream results.
Scraper packages live under internal/scrapers/ and must be blank-imported to trigger registration.
Index ¶
- func AbsentError(url string, err error) error
- func Debugf(level int, format string, args ...any)
- func HostMatches(rawURL string, hosts ...string) bool
- func LanguagesFor(sc StudioScraper) []string
- func NormalizeLanguage(s string) (string, error)
- func Pace(ctx context.Context, delay time.Duration) bool
- func Paginate(ctx context.Context, opts ListOpts, siteID string, out chan<- SceneResult, ...)
- func ParseError(url string, err error) error
- func PatternMatchesURL(pattern, rawURL string) bool
- func PreferredStudioURL(studioURL string) string
- func Register(s StudioScraper)
- func ResetWarnDelayBelow()
- func ResetWarnURLFallthrough()
- func ResolveLanguage(sc StudioScraper, requested string) (string, error)
- func SetVerbose(level int)
- func TransportError(url string, err error) error
- func URLHasNonRootPath(rawURL string) bool
- func URLLooksUnhandled(s StudioScraper, rawURL string) bool
- func Verbose() int
- func WarnDelayBelow(siteID string, actual, recommended time.Duration)
- func WarnURLFallthrough(siteID, rawURL string)
- func WithPacer(ctx context.Context, p *Pacer) context.Context
- func WorkerCount(opts ListOpts, def int) int
- type FailureKind
- type ListOpts
- type MultiLingual
- type Pacer
- type PageResult
- type ResultKind
- type SceneResult
- type ScrapeError
- type StudioScraper
- type StudioURLCanonicalizer
- type URLRecognizer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AbsentError ¶
AbsentError marks err as a resource that is legitimately not there, so it does not count against traversal completeness. Use it for optional sub-listings a site may simply not have — not as a way to silence a fetch that failed.
func Debugf ¶
Debugf prints a debug message to stderr if the current verbosity level is >= the requested level. Format follows fmt.Fprintf conventions.
Levels by convention:
1 — high-level operations (pages fetched, items found, categories discovered) 2 — HTTP requests (URL, method, status, size) 3 — parsing details (regex matches, extracted fields)
func HostMatches ¶
HostMatches reports whether rawURL's host is one of hosts, ignoring a leading "www." on either side.
It compares parsed hosts rather than searching the raw URL text, so a look-alike domain that merely contains one of hosts does not match. See CONTRIBUTING.md § "Host regexes".
func LanguagesFor ¶ added in v1.32.0
func LanguagesFor(sc StudioScraper) []string
LanguagesFor returns the languages sc accepts, or nil if it has no language dimension.
func NormalizeLanguage ¶ added in v1.32.0
NormalizeLanguage lowercases and shape-checks a language tag. An empty input returns an empty tag and no error, meaning "the scraper's own default".
func Pace ¶ added in v1.32.0
Pace blocks for this scrape's turn before the caller's next request, and reports false if ctx ended first.
With a Pacer on the context — which `fss scrape` installs from the resolved delay — the wait is shared, so concurrent workers interleave instead of bursting. Without one, it falls back to sleeping `delay` in the calling goroutine, which is what a scraper driven directly as a library does.
func Paginate ¶
func Paginate(ctx context.Context, opts ListOpts, siteID string, out chan<- SceneResult, fetchPage func(ctx context.Context, page int) (PageResult, error), )
Paginate runs a page-numbered pagination loop, handling delay, context cancellation, progress reporting, KnownIDs early-stop, and scene sending. The caller provides a FetchPage callback that fetches and parses a single page; everything else is handled by the loop.
FetchPage receives the 1-based page number and returns a PageResult. Set PageResult.Total on the first page for progress display. Set PageResult.Done to true when there are no more pages (e.g. items < pageSize, page >= totalPages). An empty Scenes slice also stops the loop, unless PageResult.Continue is set — see PageResult.Continue for pages that yield zero scenes but are not the end of the listing.
The siteID string is used only for debug log messages.
Paginate does NOT call defer close(out) — the caller's run() must still do that, since some scrapers do additional work after the pagination loop returns (e.g. worker pool teardown).
Two guards bound the loop against CMSes that never report an end: a hard page cap (paginateSafetyCap) and repeat-page detection (a page whose first scene ID matches the previous page is the CMS echoing its last page, so the loop stops). Both prevent an unbounded --full crawl when a callback forgets to set Done.
func ParseError ¶
ParseError marks err as a page that arrived but could not be understood. This is the one worth reaching for by hand: a parser that quietly returns nothing is indistinguishable from a site with nothing on it, and the difference decides whether an authoritative save may delete scenes.
func PatternMatchesURL ¶ added in v1.32.0
PatternMatchesURL reports whether rawURL is an instance of one of the display patterns a scraper advertises from Patterns().
A pattern is a host followed by an optional path and query, with `{name}` standing for one path segment or one query value — `site.com/models/{slug}`, `site.com/videos?website={id}`. Matching is on shape, not content: the host must be the same (ignoring a leading `www.`), the path must have the same number of segments with the literal ones equal case-insensitively, and every query key the pattern names must be present. A pattern segment mixing literals and placeholders (`{slug}_{page}_d.html`) matches any single segment with the same literal parts around them.
func PreferredStudioURL ¶ added in v1.30.1
PreferredStudioURL asks the scraper matching studioURL which spelling of it to use as the studio's key, so two URLs for one studio do not become two studios. A URL no scraper claims, or one whose scraper expresses no preference, is returned unchanged.
func Register ¶
func Register(s StudioScraper)
Register adds a scraper to the global registry. Call this from an init() function in each scraper package. Panics if a scraper with the same ID is already registered.
func ResetWarnDelayBelow ¶
func ResetWarnDelayBelow()
ResetWarnDelayBelow clears the once-per-siteID memoisation. Intended for tests that need to re-trigger the warning across cases.
func ResetWarnURLFallthrough ¶
func ResetWarnURLFallthrough()
ResetWarnURLFallthrough clears the once-per-key memoisation. For tests.
func ResolveLanguage ¶ added in v1.32.0
func ResolveLanguage(sc StudioScraper, requested string) (string, error)
ResolveLanguage decides which language tag a run should ask sc for; an empty result means the scraper's own default. See docs/usage.md.
func SetVerbose ¶
func SetVerbose(level int)
SetVerbose sets the global debug verbosity level. Level 0 = silent (default), 1+ = increasingly verbose.
func TransportError ¶
TransportError marks err as a page that never arrived. Scrapers rarely need this — Classify already recognises network errors and httpx status errors — but it is here for transports it cannot see into.
func URLHasNonRootPath ¶
URLHasNonRootPath reports whether rawURL carries a path (or query) component beyond the bare site root — i.e. it looks like a filtered view (model, channel, category, tag, DVD) rather than the studio's front page. A trailing "/" and the common root files ("/", "/en/", "/tour/") count as root.
Scrapers whose run() dispatches on URL shape can use this in their default (full-catalogue) branch to detect a filtered URL that fell through to scraping the entire site — see WarnURLFallthrough.
func URLLooksUnhandled ¶ added in v1.32.0
func URLLooksUnhandled(s StudioScraper, rawURL string) bool
URLLooksUnhandled reports whether rawURL names a path or query the scraper does not advertise handling.
A scraper that does not recognise a filtered URL falls through to its full catalogue and stores the result under the filtered URL's key, which looks like a successful scrape of something it is not. There is no way to ask a scraper what it recognised, but Patterns() is what it claims to accept, so a non-root URL matching none of the non-root patterns is the signal. It is advisory: a scraper may handle a form it never advertised, which is itself worth knowing.
func WarnDelayBelow ¶
WarnDelayBelow prints a one-shot stderr warning when `actual` is strictly below `recommended` for scraper `siteID`. Use it to flag when the operator's chosen delay is below a value the upstream site is known to need to avoid rate-limiting. Unlike Debugf this prints regardless of verbosity, but only once per siteID per process so it doesn't spam.
`recommended` of 0 or `actual` of 0 are no-ops if recommended <= 0; a recommended floor of zero means "no minimum", so nothing to warn about. Callers should pass the package's documented minimum (e.g. 500ms for julesjordan / newsensations).
func WarnURLFallthrough ¶
func WarnURLFallthrough(siteID, rawURL string)
WarnURLFallthrough prints a one-shot stderr warning that a filtered URL (one with a non-root path) fell through to the default full-catalogue scrape for siteID. This is the loud signal for AUDIT_PLAN S3/B3: a model/channel/tag URL that the scraper did not recognise would otherwise silently scrape the whole site under that URL's store key. It is a no-op for root URLs. Prints at most once per (siteID, path) per process.
func WithPacer ¶ added in v1.32.0
WithPacer attaches a Pacer to ctx so every goroutine of a scrape shares one schedule. The cmd layer installs it once per run from the resolved delay.
func WorkerCount ¶ added in v1.32.0
WorkerCount resolves how many parallel fetchers a scraper's detail pool should run: the operator's `--workers` when they set one, the scraper's own default otherwise.
It exists because `--workers` was silently ignored by every scraper that sized its pool from a package constant — `--workers 1` on a fragile site still ran the hardcoded four, and raising it never helped. A scraper with a reason to cap its concurrency independently of the flag (a listing walk that degrades the origin, say) should keep its own constant and say so, rather than reading the flag and ignoring it.
Types ¶
type FailureKind ¶
type FailureKind int
FailureKind classifies why a scrape step failed.
Scrapers report non-fatal failures as Error results on the channel, and for a long time every one of them was an undifferentiated `error`. Three very different situations were reaching the consumer as the same value:
- the page never arrived — the scenes on it are missing, and a retry might well get them
- the page arrived but could not be understood — the scenes on it are also missing, but the site changed and the parser needs fixing
- the requested thing is legitimately not there — nothing is missing
The distinction is load-bearing rather than cosmetic. `--full` and `--refresh` treat a traversal as authoritative over the studio's whole scene set, so an unreached page has to suppress the destructive delete, while an optional sub-listing that genuinely 404s must not — inflating the error count with expected absences made every such run non-authoritative.
const ( // FailureUnknown is a failure that has not been classified. It is the // conservative default: the plain errors scrapers already return land // here, and are treated as potentially-missing data exactly as they were // before this classification existed. FailureUnknown FailureKind = iota // FailureTransport means the page never arrived — a network error, a // timeout, a cancelled context, or a status that says the server would // not serve it (5xx, 429, 403). FailureTransport // FailureParse means the page arrived intact but could not be read: a // missing block, a selector that matches nothing, an unparseable date. // This is the site-redesign signal, and the one worth acting on — it // will not fix itself on a retry. FailureParse // FailureAbsent means the requested resource is legitimately gone (404, // 410) or the listing is genuinely empty. No data is missing, so a // traversal that only saw these is still complete. FailureAbsent )
func Classify ¶
func Classify(err error) FailureKind
Classify determines the FailureKind of an error.
It looks for an error in the chain that classifies itself (see ScrapeError), then falls back to recognising cancellation and network errors as transport failures. Anything else is FailureUnknown, which callers must treat as possibly-missing data.
A nil error is FailureUnknown; callers should not be classifying success.
func (FailureKind) MissingData ¶
func (k FailureKind) MissingData() bool
MissingData reports whether a failure of this kind means scenes went uncollected, and therefore that a traversal cannot be treated as the studio's complete state.
Everything except FailureAbsent counts, including FailureUnknown: an error nobody classified might have cost us a page, and wrongly believing a traversal complete is what deletes a catalogue.
func (FailureKind) String ¶
func (k FailureKind) String() string
String returns the lowercase name of the failure kind.
type ListOpts ¶
type ListOpts struct {
// Workers sets the number of concurrent detail-page fetchers for scrapers
// that use a worker pool. Zero uses the scraper's default (typically 4).
Workers int
// KnownIDs, when non-empty, signals the scraper to stop pagination as soon
// as it encounters an ID already in the set. Used for incremental runs where
// content is sorted newest-first and trailing pages are already stored.
KnownIDs map[string]bool
// Delay is the duration to sleep between page fetches (and between detail
// fetches for scrapers that use a worker pool). Zero means no delay.
Delay time.Duration
// Cookie is an operator-supplied Cookie header value for this site, in
// `name=value; name2=value2` form. Empty means send none.
//
// It exists for hosts that will not answer an anonymous request — an age
// gate the site sets from a button, a members area, or a bot check the
// operator's own browser has already passed. The operator satisfies the
// requirement themselves and hands the resulting cookie to fss; fss does
// not obtain, refresh or work around one. A scraper that has no use for it
// ignores it, which is all but a few of them.
Cookie string
// Language is the content language the operator asked for, as a lowercase
// tag ("de"). Empty means the scraper's own default. Resolved against
// MultiLingual before the run starts, so a scraper receives either one of
// its own languages or "".
Language string
}
ListOpts controls scraping behaviour passed in from the CLI/config.
type MultiLingual ¶ added in v1.32.0
type MultiLingual interface {
// Languages returns the tags this scraper accepts, lowercase, default first.
Languages() []string
}
MultiLingual is implemented by scrapers whose site serves the same catalogue in more than one language. See docs/usage.md.
type Pacer ¶ added in v1.32.0
type Pacer struct {
// contains filtered or unexported fields
}
Pacer spaces requests across every goroutine that shares it.
A worker pool where each goroutine sleeps `--delay` between its own requests does not pace the site at all: N workers issue N near-simultaneous requests every delay, so eight workers at the 500ms default hit ~16 requests a second rather than two. A Pacer hands out one slot per delay to whichever goroutine asks next, which is the cadence `--delay` and RecommendedDelay describe.
A zero or negative delay disables pacing entirely, matching `--delay 0`.
func NewPacer ¶ added in v1.32.0
NewPacer returns a Pacer handing out one slot every delay. A zero or negative delay yields a Pacer that never blocks.
type PageResult ¶
type PageResult struct {
Scenes []models.Scene
Total int
Done bool
// Continue tells the loop not to treat an empty Scenes slice as
// end-of-listing. Set it when a page legitimately yields zero Scenes but
// more pages may still follow — either because the page's raw items all
// filtered out (videos-only, dedup, details that failed to fetch), or
// because pagination walks a fixed list (DVDs, years) where an empty
// element is not the end. Callbacks that set Continue MUST also set Done at
// the true end, or the loop will not terminate. Leave it false for the
// common case where an empty page means the listing is exhausted.
Continue bool
}
PageResult is returned by a FetchPage callback to the pagination loop.
type ResultKind ¶
type ResultKind int
ResultKind identifies what a SceneResult carries.
const ( // KindScene indicates the result carries a valid Scene. KindScene ResultKind = iota // KindError indicates a non-fatal error. Log and continue. KindError // KindTotal is a progress hint sent once after the first page. KindTotal // KindStoppedEarly signals the scraper hit a known ID and stopped pagination. KindStoppedEarly )
func (ResultKind) String ¶
func (k ResultKind) String() string
type SceneResult ¶
type SceneResult struct {
Kind ResultKind
Scene models.Scene
Err error
Total int
}
SceneResult is a single item sent on the channel returned by ListScenes. Use the Kind field to determine which other fields are populated. Prefer the constructor functions Scene, Error, [Total], StoppedEarly.
func Error ¶
func Error(err error) SceneResult
Error constructs a SceneResult carrying a non-fatal error.
func Progress ¶
func Progress(total int) SceneResult
Progress constructs a SceneResult carrying a total-scenes hint.
func Scene ¶
func Scene(s models.Scene) SceneResult
Scene constructs a SceneResult carrying a scraped scene.
func StoppedEarly ¶
func StoppedEarly() SceneResult
StoppedEarly constructs a SceneResult signalling early pagination stop.
type ScrapeError ¶
type ScrapeError struct {
// Kind is why the step failed.
Kind FailureKind
// URL is the page being fetched or parsed, if known.
URL string
// Err is the underlying error.
Err error
}
ScrapeError annotates an error with a FailureKind and the URL it concerns, so the cmd layer can tell an unreached page from an absent one without pattern-matching on error strings.
Build one with TransportError, ParseError or AbsentError rather than filling the struct directly.
func (*ScrapeError) Error ¶
func (e *ScrapeError) Error() string
Error implements the error interface.
func (*ScrapeError) FailureKind ¶
func (e *ScrapeError) FailureKind() FailureKind
FailureKind reports how this error should be classified. It is what Classify looks for, and any error type may implement the same method to opt in.
func (*ScrapeError) Unwrap ¶
func (e *ScrapeError) Unwrap() error
Unwrap returns the underlying error so errors.Is/As see through the annotation.
type StudioScraper ¶
type StudioScraper interface {
// ID returns a stable lowercase identifier for this scraper (e.g. "manyvids").
ID() string
// Patterns returns the URL patterns this scraper handles.
// Used by `fss list-scrapers` and as documentation. A scraper may declare
// multiple patterns (different URL formats, shared-platform sites, etc.).
Patterns() []string
// MatchesURL returns true if this scraper can handle the given studio URL.
MatchesURL(url string) bool
// ListScenes fetches all scenes for the given studio URL and sends each
// result down the returned channel. The channel is closed when done.
// Implementations should respect ctx cancellation.
ListScenes(ctx context.Context, studioURL string, opts ListOpts) (<-chan SceneResult, error)
}
StudioScraper is implemented once per supported site. Adding a new site means creating internal/scrapers/<site>/<site>.go, implementing this interface, and calling scraper.Register in an init().
func ForID ¶
func ForID(id string) (StudioScraper, error)
ForID returns the registered scraper with the given ID, or an error if none match.
func ForURL ¶
func ForURL(url string) (StudioScraper, error)
ForURL returns the first registered scraper that matches the given URL, or an error if none match. Resolution is first-match-wins by registration (import) order. When more than one scraper matches — e.g. a broad parent regex shadowing a sub-site — the extra matches are reported at debug level so the overlap is visible without changing which scraper is chosen.
type StudioURLCanonicalizer ¶ added in v1.30.1
type StudioURLCanonicalizer interface {
// PreferredStudioURL returns the spelling to store studioURL under, or
// the empty string to keep it as given.
PreferredStudioURL(studioURL string) string
}
StudioURLCanonicalizer is implemented by a scraper whose site serves one studio at more than one URL. It is optional: a scraper that does not implement it keeps the operator's URL verbatim.
This is distinct from output.CanonicalStudioURL, which normalises scheme and host generically and never touches the path. Rewriting a path is only safe when the scraper asserts the two forms are the same studio.
type URLRecognizer ¶ added in v1.32.0
URLRecognizer lets a scraper vouch that it dispatches on a URL form its Patterns() cannot express. It is opt-in and structural, like FailureKind and MultiLingual: the scrapers that do not implement it are judged by their patterns alone.
The case it exists for is a table whose rows are *defined* by a URL form — a Gamma row carrying its own match regex exists precisely because that channel or studio path is the filtered view it scrapes, so MatchesURL is the authority there and a display pattern would only restate it.
Source Files