Documentation
¶
Overview ¶
Package upstream acquires the content whose license is in question: a published ecosystem artifact, or a repository at a revision.
It is the only package here that touches the network or the disk. Everything else reads an fs.FS, which is what this returns, so no other package has to know whether the tree it is reading came from a build workspace, a downloaded archive or a test fixture.
Why content and not metadata ¶
Reading a license field out of registry metadata reproduces what a build-time scanner already does, and reproduces its blind spots: Go modules have no license field anywhere in their metadata, Cargo.lock records checksums but no licenses, and a manifest that does carry a license routinely disagrees with the license files in the artifact. Metadata is worth capturing - it is the declared license, and agreement with the content raises confidence - but the conclusion has to come from classifying the content.
Identity, not trust ¶
A fetch reports whether the retrieved bytes carry the digest the caller expected, and Checksum asks the ecosystem's own checksum database the same question independently. Both are statements about content identity and neither is a statement about licensing or provenance: a match means these are the bytes that database recorded, so an answer drawn from them is safe to share with every other package depending on that version. What policy to apply to a mismatch is the caller's decision.
Posture ¶
Fetches are https-only, validated immediately before the request against the addresses the host resolves to, size-bounded, and - for artifact registries - unauthenticated, since every artifact fetched here is public by definition. A repository checkout may carry a credential, which travels in the environment and never in a command line, a log or an error.
Nothing fetched is ever executed. Archives are read for their bytes, and a build script or install hook inside one is just another file.
Index ¶
- Constants
- Variables
- func AuthEnv(ctx context.Context, repoURL string, token TokenFunc, hosts ...string) []string
- func DefaultAuthHosts() []string
- func Expand(name string, r io.ReaderAt, size int64, prefix string) (fs.FS, error)
- func RedactURL(rawURL string) string
- func Retain(p string) bool
- func ValidateURL(ctx context.Context, rawURL string) error
- type Artifact
- type CargoResolver
- type Checkout
- type ChecksumResult
- type Checksums
- type GoResolver
- type MavenResolver
- type NpmResolver
- type Option
- type PyPIResolver
- type Resolver
- type Runner
- type Set
- type TokenFunc
- type Tree
Examples ¶
Constants ¶
const ( // SumGolangOrg is the Go checksum database. Its records are what "go mod // verify" checks against, and they are transparency-log backed. SumGolangOrg = "https://sum.golang.org" // CratesIOAPI serves crates.io's own record of a version's checksum. CratesIOAPI = "https://crates.io" // MavenCentralChecksums is where Central publishes a checksum beside every // artifact. MavenCentralChecksums = MavenCentral )
The checksum databases each ecosystem publishes independently of its artifacts.
const ( // MaxExpandedBytes bounds how much an archive may expand to once its // members have been filtered down to the ones bearing on licensing. // // This is a decompression guard, not a size preference. Anyone can publish // to the registries fetched here, and a small archive that expands to // gigabytes is a known attack rather than a hypothetical one. It is // deliberately tight: after filtering, a real component's licensing // evidence is kilobytes. // MaxExpandedBytes = 64 << 20 // MaxArtifactBytes is the default bound on one staged artifact, before // expansion. // // The bound is loose because a Go module zip is a whole repository at a // revision: RoaringBitmap/roaring reaches 135 MB, azure-sdk-for-go 66 MB. // The value matches MaxDecompressedBytes, past which a tarball cannot be // expanded at all. // // A staged artifact occupies the caller's temporary directory, which is // memory on a Cloud Run service with no volume mounted. Exceeding the // bound fails the fetch and the component is reported as unknown. A caller // passes a smaller bound when its temporary directory is smaller, or when // enough fetches run at once for their sum to exceed it. MaxArtifactBytes = 1 << 30 // MaxMetadataBytes bounds a metadata response read into memory: a pom, a // checksum file, an index's JSON. These have a known modest shape, unlike // the artifacts they describe, so buffering them is reasonable and the // bound is about a hostile or broken registry rather than about size. MaxMetadataBytes = 16 << 20 // DefaultTimeout bounds how long a fetch waits without progress: for the // response headers, and thereafter between reads of the body. // // The bound is not on the whole request. http.Client.Timeout covers the // body read, which makes a time limit into a bandwidth requirement: 135 MB // in 60 seconds needs 2.25 MB/s sustained. MaxArtifactBytes bounds size. DefaultTimeout = 60 * time.Second )
const CratesIO = "https://static.crates.io"
CratesIO serves the immutable .crate archives for published crate versions.
const GoProxy = "https://proxy.golang.org"
GoProxy is the module proxy artifacts are fetched from. The proxy serves immutable, checksum-verifiable zips for every published version.
const MavenCentral = "https://repo1.maven.org/maven2"
MavenCentral serves the released artifacts for published Maven coordinates.
const ( // MaxDecompressedBytes bounds how much a tarball may decompress to in // total, members this package discards included. // // It is a separate and much looser bound than MaxExpandedBytes, which // covers only the members retained. The two are not the same guard: a tar // is read as a stream, so reaching the next member's header means // decompressing the one before it whether or not it is kept. Without this, // a gzip bomb whose members are all named payload.bin would decompress // without end while retaining nothing at all. // // The value has to clear the real artifacts. Some published archives are // genuinely large - a crate of generated bindings, a repository at a // revision - and refusing them would be refusing real dependencies, which // is the mistake an earlier download cap already made. MaxDecompressedBytes = 1 << 30 )
const NpmRegistry = "https://registry.npmjs.org"
NpmRegistry serves the immutable tarballs for published package versions.
const PyPI = "https://pypi.org"
PyPI serves the index metadata that names a release's files and their hashes. The files themselves live on files.pythonhosted.org, which the index links to.
Variables ¶
var ErrGone = errors.New("artifact permanently unavailable")
ErrGone means the registry does not and will not have this version, so retrying is pointless: the content is permanently unavailable rather than merely unread.
var ErrNotFound = errors.New("repository or ref not found")
ErrNotFound means the repository or the ref does not exist, which is a permanent answer rather than something to retry.
var ErrUnsupported = errors.New("no resolver for ecosystem")
ErrUnsupported means no resolver handles a component's ecosystem, which is a coverage gap to report rather than a failure to retry.
Functions ¶
func AuthEnv ¶
AuthEnv returns the environment variables that authenticate a clone with a minted token.
The credential travels as an Authorization header injected through git's GIT_CONFIG_PARAMETERS mechanism - the variable git itself uses to propagate -c config to child processes - so it appears in neither the process command line, which is world-readable through /proc, nor in any log. Any GIT_CONFIG_PARAMETERS already in the environment is preserved, and git resolves the last value for a single-valued key.
The config key is scoped to the specific host, so git never offers the header to another one, and the host has to be allowed before a token is minted at all.
Authentication is an enhancement here - it buys rate limit and private repositories, and public upstreams clone fine without it - so a mint failure degrades to an anonymous clone rather than failing the checkout.
It is exported because the guarantee it makes is worth testing directly.
Example ¶
AuthEnv carries a credential to git through the environment rather than the command line, and offers it only to the hosts allowed for the request.
package main
import (
"context"
"fmt"
"chainguard.dev/license/upstream"
)
func main() {
token := func(context.Context) (string, error) { return "hunter2", nil }
fmt.Println(len(upstream.AuthEnv(context.Background(), "https://github.com/example/project", token)))
fmt.Println(len(upstream.AuthEnv(context.Background(), "https://gitlab.example.com/example/project", token)))
}
Output: 1 0
func DefaultAuthHosts ¶
func DefaultAuthHosts() []string
DefaultAuthHosts returns the hosts a credential may be offered to when a request names none of its own.
An allowlist rather than a policy of sending the token wherever the URL points: the token is a GitHub token, and a URL chosen by something untrusted must not be able to make git present it to a host of the attacker's choosing. It is a function returning a fresh slice rather than a variable, so that the allowlist cannot be widened process-wide by anything that imports this package.
func Expand ¶
Expand reads an archive into a filesystem, choosing the format from the filename.
It is exported for the caller holding an archive this package did not fetch: a published jar or wheel handed to it directly, or an apk downloaded by a backfill sweep. prefix is the single directory the archive nests its contents under, or empty when it has none.
func RedactURL ¶
RedactURL removes any userinfo from a URL so it can be logged or returned in an error.
Validation rejects a credentialed URL outright, so this is for the caller that has to name the URL it was given - in a log line, a trace, or a tool result - before or regardless of whether validation accepted it.
Example ¶
RedactURL is for the caller that has to name the URL it was given in a log line or a tool result.
package main
import (
"fmt"
"chainguard.dev/license/upstream"
)
func main() {
fmt.Println(upstream.RedactURL("https://user:secret@github.com/example/project"))
}
Output: https://github.com/example/project
func Retain ¶
Retain reports whether a file inside a published artifact can bear on licensing, and is therefore worth expanding.
The point is to expand only what licensing needs. Some published artifacts are enormous - the windows crate ships tens of megabytes of generated bindings, which exceeded the expansion limit and cost the whole component -- while the evidence in them is a handful of kilobyte-sized files. Retaining that handful turns an artifact that could not be read at all into one that can.
The retained set is a superset of what classification reads, so filtering can never drop a text that would have been classified: license files by the shared discovery rule, plus the manifests a declared license is read from and the prose files a relationship statement is read from.
What is lost is the license headers in source files, which a deeper analysis could otherwise have read. That is a smaller loss than having no content at all.
Example ¶
Retain is the filter that makes large artifacts readable at all: some crates ship tens of megabytes of generated bindings, while the licensing evidence in them is a handful of kilobyte-sized files.
package main
import (
"fmt"
"chainguard.dev/license/upstream"
)
func main() {
for _, path := range []string{"LICENSE", "Cargo.toml", "README.md", "src/generated.rs"} {
fmt.Printf("%-18s %v\n", path, upstream.Retain(path))
}
}
Output: LICENSE true Cargo.toml true README.md true src/generated.rs false
func ValidateURL ¶
ValidateURL rejects URLs this package must not fetch.
Upstreams are open-world - any public code host, any public registry - so rather than an allowlist this blocks the internal targets a URL chosen by something untrusted could aim at: non-https schemes, embedded userinfo credentials, IP-literal hosts such as link-local metadata services, localhost, and internal-only DNS suffixes.
String checks alone are not enough, because they miss the non-standard IP spellings an OS resolver accepts (127.1, 0x7f000001) and DNS-rebinding hostnames (127.0.0.1.nip.io). So the host is resolved and every address it resolves to has to be public. Resolution failure rejects, which is to say the check fails closed.
It is called immediately before each request rather than once when a URL is accepted, because a name that resolved publicly a minute ago can resolve somewhere else now. There is deliberately no way to fetch through this package without it.
Example ¶
ValidateURL refuses the internal targets a URL chosen by something untrusted could aim at. There is no way to fetch through this package without it.
package main
import (
"context"
"fmt"
"chainguard.dev/license/upstream"
)
func main() {
for _, url := range []string{
"http://proxy.golang.org/example.com/@v/v1.0.0.zip",
"https://169.254.169.254/latest/meta-data",
"https://metadata.google.internal/computeMetadata/v1/",
"https://user:secret@github.com/example/project",
} {
fmt.Println(upstream.ValidateURL(context.Background(), url))
}
}
Output: unsupported URL scheme: only https is allowed, got "http://proxy.golang.org/example.com/@v/v1.0.0.zip" URL "https://169.254.169.254/latest/meta-data": IP-literal hosts are not allowed URL "https://metadata.google.internal/computeMetadata/v1/": internal hosts are not allowed URL must not contain userinfo credentials
Types ¶
type Artifact ¶
type Artifact struct {
// FS is a read-only filesystem over the archive's contents, rooted at the
// component itself.
FS fs.FS
// Digest is what the retrieved bytes actually hash to, in the ecosystem's
// own scheme.
Digest string
// DigestAlgo names that scheme.
DigestAlgo string
// Matched reports that Digest is the digest the caller said to expect,
// which for most ecosystems is what a lockfile recorded. False either
// because they differed or because there was no expected digest to compare
// against, so a caller that needs to tell those apart checks whether it
// supplied one.
Matched bool
// Source is the URL the content came from.
Source string
// Notes record anything about the fetch a reader should know: a jar that
// was not published, a parent POM that could not be fetched.
Notes []string
}
Artifact is the fetched content of one component version.
type CargoResolver ¶
type CargoResolver struct {
// contains filtered or unexported fields
}
CargoResolver retrieves published crate content.
Cargo.lock records a crate's name, version and sha256 checksum but no license: the license field lives in the crate's Cargo.toml, which the lockfile does not contain. Fetching the .crate archive recovers both - the declared license from its Cargo.toml, and the concluded license from the license files it ships.
func NewCargoResolver ¶
func NewCargoResolver(client *http.Client, opts ...Option) *CargoResolver
NewCargoResolver returns a resolver reading from crates.io.
func (*CargoResolver) Ecosystem ¶
func (r *CargoResolver) Ecosystem() component.Ecosystem
Ecosystem implements Resolver.
type Checkout ¶
type Checkout struct {
// RepoURL is the full https URL of the repository.
RepoURL string
// Ref is the tag or branch to check out. A tag is the usual case: the
// license of a version is whatever that version shipped, so a floating
// branch answers a different question.
Ref string
// Token authenticates the clone. It is optional, and is offered only to
// the hosts AuthHosts allows.
Token TokenFunc
// AuthHosts are the hosts a credential may be offered to. Empty means
// DefaultAuthHosts.
AuthHosts []string
// Run observes the git operation, defaulting to just performing it.
Run Runner
}
Checkout describes a repository revision to acquire.
type ChecksumResult ¶
type ChecksumResult struct {
// Recorded is the digest the database holds for this version, empty when
// the database has no record of it.
Recorded string
// Matched reports that Recorded equals the digest the caller asked about.
Matched bool
// Source is the URL the record came from, so a result can be re-checked by
// hand.
Source string
}
ChecksumResult is what an ecosystem's checksum database says about a component version.
type Checksums ¶
type Checksums struct {
// contains filtered or unexported fields
}
Checksums queries the checksum databases the ecosystems publish.
func NewChecksums ¶
NewChecksums returns a client reading from the public databases.
func (*Checksums) Lookup ¶
Lookup asks the ecosystem's own checksum database what digest it recorded for a component version, and whether the digest the caller holds matches.
This is a second, independent witness, which is the whole reason the check exists. A digest computed from bytes a registry served says only that the download arrived intact; the same registry could serve different bytes tomorrow. A digest that also matches what the ecosystem's own database recorded says these are the bytes the ecosystem published, so an answer drawn from them is safe to share with every package depending on that version.
What a match does not say: anything about licensing, and nothing about provenance beyond content identity. Each caller decides what its own policy requires of the result.
An ecosystem with no independent database returns ErrUnsupported. PyPI and npm are in that position: their digests come from the same service as the bytes, so asking again corroborates nothing.
type GoResolver ¶
type GoResolver struct {
// contains filtered or unexported fields
}
GoResolver retrieves published Go module content.
Go is the ecosystem with the largest license gap and the one where metadata cannot help at all: there is no license field in go.mod, so nothing short of reading the module's own files can answer the question. The h1 hash recorded in go.sum identifies exactly the bytes the proxy serves, which is what makes a determination drawn from them shareable with every package depending on that version.
func NewGoResolver ¶
func NewGoResolver(client *http.Client, opts ...Option) *GoResolver
NewGoResolver returns a resolver reading from the default module proxy.
func (*GoResolver) Ecosystem ¶
func (r *GoResolver) Ecosystem() component.Ecosystem
Ecosystem implements Resolver.
type MavenResolver ¶
type MavenResolver struct {
// contains filtered or unexported fields
}
MavenResolver retrieves published Java artifact content.
Java differs from the other ecosystems in where its licensing lives. Only 44 of the 109 jars in a representative package carry a license file at all, so the jar is a weak source of a conclusion; the POM's <licenses> block is the strong one. That block is also frequently inherited: of the 65 POMs in that same package, 29 declare licenses directly and 36 declare them only through a <parent>, so following the chain is most of the coverage rather than a refinement of it.
Both are fetched. The jar supplies whatever license files it ships, and the POM chain supplies the declaration, written into the returned filesystem so the declared-license reader finds it the same way it finds any other manifest.
func NewMavenResolver ¶
func NewMavenResolver(client *http.Client, opts ...Option) *MavenResolver
NewMavenResolver returns a resolver reading from Maven Central.
func (*MavenResolver) Ecosystem ¶
func (r *MavenResolver) Ecosystem() component.Ecosystem
Ecosystem implements Resolver.
type NpmResolver ¶
type NpmResolver struct {
// contains filtered or unexported fields
}
NpmResolver retrieves published npm package content.
npm is the one ecosystem here where the lockfile is a genuinely independent witness to the content. It records a Subresource Integrity hash written when the dependency was first installed, so comparing the registry's bytes against it verifies the two agree rather than just verifying the registry against itself. That is the same footing Go and Rust are on, and stronger than what PyPI or Maven Central can offer.
func NewNpmResolver ¶
func NewNpmResolver(client *http.Client, opts ...Option) *NpmResolver
NewNpmResolver returns a resolver reading from the public registry.
func (*NpmResolver) Ecosystem ¶
func (r *NpmResolver) Ecosystem() component.Ecosystem
Ecosystem implements Resolver.
type Option ¶
type Option func(*options)
Option tunes a resolver for the environment it runs in.
func WithMaxArtifactBytes ¶
WithMaxArtifactBytes bounds one staged artifact, in place of MaxArtifactBytes.
A caller sets it when its temporary directory is tighter than the default assumes, or when it fetches enough artifacts at once that the product is what matters: the bound times the number in flight is what has to fit, and only the caller knows the second term.
type PyPIResolver ¶
type PyPIResolver struct {
// contains filtered or unexported fields
}
PyPIResolver retrieves published Python distribution content.
Python components are enumerated from what the build installed, which records a name and a version but no hash: an installed tree is unpacked files, not the archive they came from. The index supplies both the archive and the hash it recorded for it, so the determination is keyed to the release every other package installing this dependency also gets, rather than to one rebuild.
func NewPyPIResolver ¶
func NewPyPIResolver(client *http.Client, opts ...Option) *PyPIResolver
NewPyPIResolver returns a resolver reading from the public index.
func (*PyPIResolver) Ecosystem ¶
func (r *PyPIResolver) Ecosystem() component.Ecosystem
Ecosystem implements Resolver.
type Resolver ¶
type Resolver interface {
// Ecosystem is the purl type this resolver handles.
Ecosystem() component.Ecosystem
// Fetch retrieves the component's published artifact. A component whose
// content is permanently gone from the registry returns an error wrapping
// ErrGone; anything else the caller may retry.
Fetch(ctx context.Context, c component.Component) (*Artifact, error)
}
Resolver retrieves the published content of components in one ecosystem.
type Runner ¶
Runner observes a git operation.
It exists so that a caller in an instrumented environment keeps its instrumentation. A caller that emits a log line and a metric per git operation would otherwise lose both for every license checkout, because this package shells out itself. A runner is handed the operation's name and a function that performs it, and it must call that function.
It is deliberately not handed the command. The environment carries the credential, so a hook that could read it could log it, and the invariant that a credential never reaches a log should not rest on every caller's hook being careful.
type Set ¶
type Set struct {
// contains filtered or unexported fields
}
Set dispatches components to the resolver for their ecosystem.
func DefaultSet ¶
DefaultSet returns the resolver set for every ecosystem with content retrieval implemented, reading from the public registries.
Example ¶
A resolver set fetches the published content of a component so that its license can be concluded from the bytes upstream actually shipped. The component's own digest, from a lockfile, is what the fetch is checked against.
package main
import (
"context"
"fmt"
"chainguard.dev/license/component"
"chainguard.dev/license/upstream"
)
func main() {
resolvers := upstream.DefaultSet(nil)
fmt.Println(len(resolvers.Ecosystems()), "ecosystems can be fetched")
// An ecosystem with no resolver is a coverage gap to report rather than a
// failure to retry.
_, err := resolvers.Fetch(context.Background(), component.Component{
Ecosystem: "nuget",
Name: "Newtonsoft.Json",
Version: "13.0.3",
})
fmt.Println(err)
}
Output: 5 ecosystems can be fetched no resolver for ecosystem: nuget
func NewSet ¶
NewSet builds a resolver set. A later resolver for the same ecosystem replaces an earlier one, so a caller can substitute one without rebuilding the rest.
func (*Set) Ecosystems ¶
Ecosystems lists the ecosystems the set can retrieve content for. A caller measuring coverage needs to distinguish a component nobody looked at from one this module cannot yet fetch.
type TokenFunc ¶
TokenFunc mints a credential for authenticating a checkout. Returning an error degrades to an anonymous clone rather than failing the checkout, since public upstreams clone fine without one.
The module takes a function rather than an OAuth token source so that it depends on no particular credential library, and so that a token is minted at the moment of use rather than held.
type Tree ¶
type Tree struct {
// FS is the checkout's contents. Path resolution is bounded to the
// checkout, so a symlink inside it that points outside fails to resolve
// rather than reading the host filesystem.
FS fs.FS
// Ref and RepoURL are what was checked out.
Ref string
RepoURL string
// contains filtered or unexported fields
}
Tree is a checked-out repository.
func CheckoutRef ¶
CheckoutRef shallow-clones a repository at a ref and returns a filesystem over it.
The URL is validated immediately before the clone, so there is no way to reach this without the host checks in ValidateURL. The clone is depth-1 and single-branch: it fetches the tree at the revision whose license is in question and nothing else.
The returned filesystem is rooted with os.Root, which bounds path resolution to the checkout. Repositories commonly symlink LICENSE to the real text, so links have to be followed; bounding resolution is what keeps a link pointing at /etc from being read as a license.
Source Files