Documentation
¶
Overview ¶
Package brick is the two edges of the container, as directories, with no Docker code in reach.
An image becomes a brick as soon as it honours the contract: it is given an envelope on standard input and under /agk/in/<port>/, and what it leaves under /agk/out/ is collected. Nothing else is promised, no daemon, no callback and no socket, and this package knows the contract and nothing about how a container is started. That is what lets agk brick test run a brick against sample envelopes with no driver in reach.
WriteInputs materialises what a step is given, Collect reads back what it produced, and Spill is the other half of the size threshold: it moves a value the engine itself constructed above inline_max_bytes into the store and replaces it with a files[] entry.
Index ¶
- Constants
- func Collect(dir string, declared []agk.Port, m agk.Meta, l agk.Limits) (map[agk.Port]agk.Envelope, error)
- func Spill(ctx context.Context, s Putter, e agk.Envelope, l agk.Limits) (agk.Envelope, error)
- type Manifest
- type ManifestMetadata
- type ManifestPort
- type ManifestResources
- type ManifestSpec
- type Mount
- type Param
- type Putter
- type Runtime
- type Secret
Constants ¶
const ( Root = "/agk" InDir = "/agk/in" OutDir = "/agk/out" OutPortsDir = "/agk/out/ports" OutFilesDir = "/agk/out/files" )
The paths the contract names. A brick reads its inputs under InDir and writes its outputs under OutDir, and those two directories are the whole of what is promised.
const ManifestPath = "/agk/brick.yaml"
ManifestPath is where a brick carries its manifest inside its image. An image becomes a brick by carrying this file, and the file is the brick's own half of the contract: what it reads, what it writes, what it needs and what account it runs as.
Variables ¶
This section is empty.
Functions ¶
func Collect ¶
func Collect(dir string, declared []agk.Port, m agk.Meta, l agk.Limits) (map[agk.Port]agk.Envelope, error)
Collect reads back what a step produced: one envelope for every output port it declares, taken from <port>.json under /agk/out/ports/.
A declared port the container never wrote publishes an empty envelope, and that is success rather than an error. It is what lets a brick write only the port it has something to say on: a step that rejected nothing leaves rejected.json unwritten, the edge below still reads a batch of nothing, and the step downstream decides its own fate through when. A brick that writes nothing at all under /agk/out/ports/ and exits 0 therefore publishes an empty envelope on each of its ports, which is the whole of what it owes the contract.
dir is the host side of /agk/out. What a container left under files/ is not read here: an artifact is uploaded and referenced by whatever holds the store, and the standard output shorthand for the single out port belongs there too, since only the caller that started the container captured the stream.
m carries what the runner knows about the batch and the container does not decide: the run, the step, the attempt and the publication time. m.Port is not read, because the port is the name of the file the envelope was found in.
The metadata a container writes is the metadata it was given. run_id, step, port and attempt reach it as AGK_RUN_ID, AGK_STEP, the file it chose to write and AGK_ATTEMPT, so an envelope disagreeing on any of them is not the batch that belongs on this port and is refused rather than quietly corrected. produced_at is the one member stamped here: a port is published when the emitting step ends, and that is a moment only the runner is in a position to know.
func Spill ¶
Spill writes any value of data heavier than inline_max_bytes to the store and replaces it with a files[] entry.
Above inline_max_bytes a value must be written as an artifact and referenced in files[], and the runner rejects an envelope that carries one inline. This is the other half of that threshold: where the runner refuses what a container emitted, Spill moves a value the engine itself constructed, so that the engine never emits an envelope its own rule would reject.
The unit is a top level field of data, because that is the unit a port's schema names and the unit a consumer asks for by name. agk refuses at a finer grain, naming the most specific value above the threshold, so that a step is told which of its fields to write as an artifact; what is moved here is the field carrying that value, and the envelope that comes back carries nothing above the threshold either way.
What is measured is the JSON encoding of the value, which is exactly what would have travelled inline, and it is those same bytes that are stored. The field leaves data and the artifact arrives in files[] of the same item, so the item keeps its identity and its rank.
Items and files already carried are left as they are: a files[] entry is already a reference, and nothing is spilled twice.
Types ¶
type Manifest ¶
type Manifest struct {
APIVersion string
Kind string
Metadata ManifestMetadata
Spec ManifestSpec
// contains filtered or unexported fields
}
Manifest is /agk/brick.yaml as a value.
It is read here rather than in the evaluator because it is the brick's half of the contract and this package already is the two edges of the container. A second reader in the evaluator would be a second answer to what a manifest is, and agk brick test gets this one with no evaluator in reach.
What this type applies is the manifest's own rules: the five required keys, the closed blocks wherever the document is the engine's and the open ones wherever the author is writing JSON Schema, the name grammar, the version grammar, a secret mounted under /agk/secrets/ and a CPU request written as a string. The rules that hold a manifest and a workflow together, that a step's outputs are a subset of the ports declared here and that its params satisfy the schemas declared here, stay in the evaluator, which is where a refusal can name the step.
func ParseManifest ¶
ParseManifest reads /agk/brick.yaml and applies the manifest's own rules.
The document is read with a YAML 1.2 parser and then read by hand, closed, on the same ground the envelope is: a manifest that does not hold together has to be told what is wrong with it in the words the documentation uses, and a JSON Schema error names a keyword and a JSON Pointer. The blocks the engine owns are closed, so "an unknown key on the root, on metadata, on spec, on a port entry, on runtime, on resources or on a secret entry fails publication rather than being ignored". The places where the author is writing JSON Schema stay open and travel through untouched: a params entry, a definitions entry and a port schema "carry the whole 2020-12 vocabulary".
It is the reader of a manifest a version is about to be made with, so it holds a port to agk.PortMaxBytes as well (CheckPortLengths). A manifest a version already holds is read with ParseStoredManifest instead.
func ParseStoredManifest ¶ added in v0.4.0
ParseStoredManifest is ParseManifest less the bound a port is written to, agk.PortMaxBytes: the reader of a manifest a version already holds, which is every manifest a version is rebuilt from and every one a runner reads out of the image a task names.
A version recorded before the bound was may hold a brick declaring a port of 251 to 255 characters, and its runs, its replays and the rebuild of its graph have to go on as they did before the upgrade: every rule stands, the grammar and the 255 characters no filesystem goes past included, and this one is refused where a version is made rather than where one is read back.
func (Manifest) CheckPortLengths ¶ added in v0.4.0
CheckPortLengths refuses a port declared past agk.PortMaxBytes, inputs and outputs alike.
An output port becomes the file <name>.json under /agk/out/ports/, which a filesystem holds to 255 characters with its suffix, and an input port is the port an edge of the workflow feeds, which the workflow names within the same bound. It is what ParseManifest adds to ParseStoredManifest, and it stands apart so that a manifest already read, and kept by image digest, can be held to it where a version is about to be made.
func (Manifest) Document ¶
Document is the manifest as a JSON document: the bytes a #/spec/definitions/<name> pointer resolves against.
"A pointer inside a manifest is a JSON Pointer resolved against the manifest document itself", so a port schema written as { $ref: "#/spec/definitions/order" } can only be compiled with the whole manifest in hand. One difference from the file as it was written is recorded here rather than left to be discovered: a parameter's boolean required is removed, because it is the language's keyword and not JSON Schema's, and a document keeping it does not compile.
func (Manifest) InputPorts ¶
InputPorts are the ports the brick reads, in name order.
func (Manifest) OutputPorts ¶
OutputPorts are the ports the brick writes, in name order.
func (Manifest) Port ¶
func (m Manifest) Port(name agk.Port) (ManifestPort, bool)
Port answers with the port the manifest declares under that name, an input before an output. The two blocks are separate in the contract, one mounted under /agk/in/ and one collected from /agk/out/ports/, so a name written in both is one brick reading and writing the same thing and either answer is the same schema.
type ManifestMetadata ¶
ManifestMetadata is what the brick is called and which release this is.
type ManifestPort ¶
type ManifestPort struct {
Schema json.RawMessage
}
ManifestPort is one declared port. It carries a schema and nothing else, and the schema itself is optional: "a port declared with no schema accepts whatever arrives, which is what a passthrough brick says".
Schema is the JSON Schema document as it was written, not a compiled one. Compiling it is the evaluator's business and needs a JSON Schema implementation, which this package does not carry: keeping the bytes is what lets the manifest be read anywhere.
type ManifestResources ¶
ManifestResources is what the brick asks for, before the runner policy and the namespace quota cap it.
type ManifestSpec ¶
type ManifestSpec struct {
Inputs map[agk.Port]ManifestPort
Outputs map[agk.Port]ManifestPort
Params map[string]Param
Secrets []Secret
Runtime Runtime
Resources ManifestResources
Definitions map[string]json.RawMessage
}
ManifestSpec is what the brick reads, what it writes, and what it needs from its container.
type Mount ¶
Mount is one input directory, as the driver has to bind it.
ReadOnly is a stated field rather than something the driver is left to remember: an input a step can write to is an input a retry of that step reads differently, and the contract says the mount is read-only.
func WriteInputs ¶
func WriteInputs(ctx context.Context, s *artifact.Store, dir string, in map[agk.Port]agk.Envelope) ([]Mount, error)
WriteInputs materialises what a step is given, under dir, and returns the mounts the driver binds read-only at /agk/in/<port>/.
Each port gets a directory holding envelope.json and, beside it, the bytes of every files[] entry under its own name. The brick therefore opens a path and never a store URL: the agk:// URI stays in the envelope, logical and unfetchable from inside the container, which is what keeps the container off the object store and off the network it would need to reach one.
The directory a file lands in is named by the key of in, which is the port of the consuming step, not meta.port of the envelope that travelled: a step reads what arrives on its own input port, whatever the port it was emitted from was called. A refusal names that input port and the step the envelope came from, which is the pair that locates the fault: an envelope one step wrote, arriving where another reads it.
type Param ¶
type Param struct {
Schema json.RawMessage
Required bool
Sensitive bool
}
Param is one declared parameter: a JSON Schema subschema, and the one keyword that is not JSON Schema's.
Required is required: true written on the parameter itself, saying that a step must supply it. The array form of the key keeps its JSON Schema meaning and stays in Schema, where it names the members the value must carry; only the boolean is the language's own, and it is lifted out of Schema so that what is left is a document a JSON Schema implementation will compile.
type Putter ¶ added in v0.2.0
type Putter interface {
Put(ctx context.Context, u agk.URI, mediaType string, r io.Reader) (agk.File, error)
}
Putter is what a spilled value is written through: an *artifact.Store, or anything that answers a write with the files[] entry the store would answer.
It is an interface rather than the store itself so that a caller can hold the envelope Spill returns to the size rules before anything reaches the store. A runner refuses an envelope before its first upload, and the entries that envelope carries are the ones the store will answer, so it describes the writes here and makes them once the envelope has passed.