Documentation
¶
Overview ¶
Package jobsync is a background job system with a Hangfire-style API and pluggable storage. Postgres, MySQL and Redis drivers are maintained in this repository; the Storage contract in storage.go is small enough that a new driver is a weekend, not a month.
Index ¶
- Variables
- func AllowAll(http.ResponseWriter, *http.Request) bool
- func Dashboard(store Storage, cfg DashboardConfig) http.Handler
- func LoopbackOnly(w http.ResponseWriter, r *http.Request) bool
- func ValidateCron(spec string) error
- type Authorizer
- type Bucket
- type Client
- type ClientOption
- type Counts
- type DashboardConfig
- type Filter
- type Heartbeat
- type Job
- type JobType
- func (j *JobType[T]) Cron(ctx context.Context, c *Client, id, spec string, args T, opts ...Option) error
- func (j *JobType[T]) Enqueue(ctx context.Context, c *Client, args T, opts ...Option) (string, error)
- func (j *JobType[T]) EnqueueTx(ctx context.Context, c *Client, tx Tx, args T, opts ...Option) (string, error)
- func (j *JobType[T]) Handle(srv *Server, fn func(context.Context, T) error)
- func (j *JobType[T]) Kind() string
- func (j *JobType[T]) Schedule(ctx context.Context, c *Client, args T, at time.Time, opts ...Option) (string, error)
- type Locker
- type Monitor
- type Option
- type QueueControl
- type QueueStat
- type Result
- type Schedule
- type Schedules
- type Server
- type ServerConfig
- type ServerInfo
- type State
- type Storage
- type Transition
- type Tx
- type TxEnqueuer
Constants ¶
This section is empty.
Variables ¶
var ErrPermanent = errors.New("jobsync: permanent failure")
ErrPermanent marks a failure that retrying cannot fix. A handler returning an error wrapping it sends the job straight to StateDead, skipping its remaining attempts.
The canonical case is a payload that will not decode: it will not decode on the tenth attempt either, so the retries are pure waste and they bury the real failures under noise. Use it for anything the job itself can never resolve — a malformed payload, a deleted row, a permanently rejected request.
var ErrUnsupportedFilter = errors.New("jobsync: storage cannot serve this filter")
ErrUnsupportedFilter is returned by Monitor.ListJobs for a Filter field the driver cannot serve. Filter.Search is the usual one: Postgres answers it with a trigram index, Redis cannot answer it at all without indexing every job's text. The dashboard disables the control rather than showing wrong results.
State, Queue and Kind filters are mandatory for every driver; they are a secondary index, not a search.
Functions ¶
func AllowAll ¶
func AllowAll(http.ResponseWriter, *http.Request) bool
AllowAll disables authorization. Name it explicitly in the call so that an open dashboard is always a decision somebody made, and is greppable.
func Dashboard ¶
func Dashboard(store Storage, cfg DashboardConfig) http.Handler
Dashboard serves the UI and its JSON API. It needs a driver implementing Monitor; with any other driver it reports that this storage has no introspection rather than failing at startup, so a dashboard route does not stop an application from booting.
func LoopbackOnly ¶
func LoopbackOnly(w http.ResponseWriter, r *http.Request) bool
LoopbackOnly admits requests that originate on this machine. It is the default Authorizer: a dashboard needs no configuration during development and is not silently world-readable once deployed.
A forwarding header means a proxy is in front, and the proxy itself is very often on loopback — which would otherwise make this wide open to the internet. Such requests are refused: a dashboard behind a reverse proxy must declare a real Authorizer.
func ValidateCron ¶
ValidateCron reports whether spec is a usable cron expression. Callers that build specs from configuration should use it at startup rather than finding out at the first fire.
Types ¶
type Authorizer ¶
type Authorizer func(w http.ResponseWriter, r *http.Request) bool
Authorizer reports whether r may see the dashboard. It takes the ResponseWriter so an implementation can send a challenge — BasicAuth needs to write WWW-Authenticate or the browser never prompts. When it returns false it owns the response; the dashboard writes nothing further.
func AuthorizeFunc ¶
AuthorizeFunc adapts an existing session or SSO check, e.g. one that reads a cookie and redirects to a login page.
func BasicAuth ¶
func BasicAuth(username, password string) Authorizer
BasicAuth gates the dashboard on HTTP basic credentials. Comparison is constant-time over digests, so neither the value nor the length of the real credentials leaks through response timing.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client enqueues jobs. It is safe for concurrent use and holds no state beyond the storage handle, so an application typically constructs exactly one.
type ClientOption ¶
type ClientOption func(*Client)
ClientOption configures a Client at construction.
func InLocation ¶
func InLocation(loc *time.Location) ClientOption
InLocation sets the timezone cron expressions are evaluated in.
The default is UTC, not time.Local. A schedule stores its zone by name, and time.Local is named the literal string "Local" — which every server then resolves to its OWN local zone. One fleet with mixed TZ settings would fire the same nightly job at several different hours, and each server would look correct in isolation.
Pass a real zone when business hours matter:
jobsync.InLocation(mustLoad("Africa/Lagos"))
type Counts ¶
type Counts struct {
Scheduled, Enqueued, Running, Retrying, Succeeded, Dead, Cancelled int64
}
Counts is a tally per state. A struct rather than a map because the state set is closed and the dashboard reads every field.
type DashboardConfig ¶
type DashboardConfig struct {
// BasePath is where the dashboard is mounted, e.g. "/jobs". Links are built
// relative to it, so it must match the mux pattern.
BasePath string
// Auth gates every request. Leaving it nil is the right thing on a laptop and
// safe on a server: the dashboard then serves loopback clients only. Set it to
// BasicAuth for a deployment, or AllowAll to deliberately open it up.
Auth Authorizer
}
DashboardConfig configures the dashboard handler.
type Filter ¶
type Filter struct {
States []State
Queues []string
Kinds []string
Tags []string
Search string // substring match over Kind, ID and LastError
Since time.Time
Until time.Time
Offset int
Limit int
}
Filter is the dashboard's query. Zero fields mean no constraint on that field; Limit of 0 means the driver's default page size.
type Heartbeat ¶
type Heartbeat interface {
Heartbeat(ctx context.Context, info ServerInfo, ttl time.Duration) error
}
Heartbeat records that a server is alive and what it is working on. Servers call it periodically; Monitor.Servers reads back the ones still within ttl.
A driver may skip it, and then the dashboard's server list is empty rather than wrong. Nothing about running jobs depends on it — it is observability, not coordination, and in particular it is NOT how abandoned work is recovered. That is Reclaim, driven by lease expiry, which keeps working whether or not anyone is writing heartbeats.
type Job ¶
type Job struct {
ID string
Kind string // handler name, from Declare
Queue string
Payload []byte // JSON-encoded args
Priority int // lower runs first
Attempt int // completed attempts; 0 until the first failure
MaxAttempts int
State State
UniqueKey string // empty means no deduplication
Tags []string
LastError string
CreatedAt time.Time
ScheduledAt time.Time // when the job becomes fetchable
LeasedUntil time.Time // meaningful only in StateRunning
FinishedAt time.Time // zero until the job reaches a terminal state
Owner string // server ID holding the lease
}
Job is the unit of work as it is persisted. Drivers own the storage layout but must be able to round-trip every field here.
type JobType ¶
type JobType[T any] struct { // contains filtered or unexported fields }
JobType is a typed handle to one kind of job. Declare it at package level; it carries no dependencies, so it can be referenced from anywhere that enqueues:
var SendWelcome = jobsync.Declare[WelcomeArgs]("email.welcome")
The implementation is bound separately, at wiring time, where dependencies actually exist:
SendWelcome.Handle(srv, func(ctx context.Context, a WelcomeArgs) error {
return mailer.Send(ctx, a.Email)
})
That split is the point. Hangfire resolves the job target from an IoC container at execution time; Go has no ambient container, and the usual answer — an args type with a Kind() method plus a registered worker struct — makes you write three declarations for one job. Here the enqueue side needs no dependencies and the handler side is a closure over them.
func Declare ¶
Declare creates a job type. kind is the name persisted with every job, so it must stay stable across deploys: renaming it strands jobs already in storage.
func (*JobType[T]) Cron ¶
func (j *JobType[T]) Cron(ctx context.Context, c *Client, id, spec string, args T, opts ...Option) error
Cron registers a recurring job. id must be stable across deploys; re-running with the same id updates the schedule in place rather than duplicating it.
func (*JobType[T]) Enqueue ¶
func (j *JobType[T]) Enqueue(ctx context.Context, c *Client, args T, opts ...Option) (string, error)
Enqueue queues args for execution as soon as a server picks it up.
func (*JobType[T]) EnqueueTx ¶
func (j *JobType[T]) EnqueueTx(ctx context.Context, c *Client, tx Tx, args T, opts ...Option) (string, error)
EnqueueTx queues args inside the caller's transaction, so the job and the rows that justify it commit together. It fails if the driver is not a TxEnqueuer — loudly, because silently falling back to a separate write would reintroduce exactly the race the caller is avoiding.
func (*JobType[T]) Handle ¶
Handle binds the implementation of j on srv. Calling it twice for the same kind panics: a duplicate registration means one of the two handlers would silently never run.
type Locker ¶
type Locker interface {
Lock(ctx context.Context, key string, ttl time.Duration) (token string, err error)
Unlock(ctx context.Context, key, token string) error
}
Locker is required for recurring jobs: it elects one server to fire a given schedule. A driver without it can still run one-off jobs; Server.Run refuses to start if schedules are registered and the driver is not a Locker.
Lock returns an empty token when the lock is held elsewhere, which is the normal case, not an error. Unlock must verify the token so a server whose lease already expired cannot release the lock a different server now holds.
type Monitor ¶
type Monitor interface {
Counts(ctx context.Context) (Counts, error)
QueueStats(ctx context.Context) ([]QueueStat, error)
Servers(ctx context.Context) ([]ServerInfo, error)
// ListJobs returns a page of jobs and the total matching f.
ListJobs(ctx context.Context, f Filter) ([]*Job, int64, error)
JobHistory(ctx context.Context, id string) (*Job, []Transition, error)
// Throughput returns succeeded and failed counts bucketed over [since, now].
// Drivers that keep no history may return nil; the dashboard hides the graph.
//
// The window is bounded by whatever retention the driver is configured with:
// a job deleted by the janitor is no longer counted. Asking for a year of
// throughput from a store that keeps seven days of jobs returns seven days.
Throughput(ctx context.Context, since time.Time, bucket time.Duration) ([]Bucket, error)
// Requeue moves jobs back to StateEnqueued with Attempt reset. Delete removes
// them outright. Both are no-ops for jobs currently in StateRunning.
Requeue(ctx context.Context, ids []string) error
Delete(ctx context.Context, ids []string) error
}
Monitor backs the dashboard. It is one interface rather than a read half and a write half because the UI needs all of it to be useful: implement it and every panel works, skip it and the dashboard reports that this driver has no introspection.
type Option ¶
type Option func(*Job)
Option adjusts a job at enqueue time. Every field it touches has a working default, which is why these are options and not parameters.
Pass them to Enqueue, Schedule, EnqueueTx or Cron:
SendWelcome.Enqueue(ctx, client, args,
jobsync.Queue("mail"),
jobsync.MaxAttempts(5),
)
func MaxAttempts ¶
MaxAttempts caps how many times the job runs before it is marked dead. Default 10. A handler returning an error wrapping ErrPermanent skips the remaining attempts regardless.
func Priority ¶
Priority orders the job within its queue. Lower runs first; default 0. Honoured exactly by the SQL drivers, and clamped to 0..9 by Redis, where a sorted-set score cannot encode both priority and due time.
func Queue ¶
Queue routes the job to a named queue. Default: "default". A server only runs the queues it was configured with, so this is how work is partitioned across pools.
type QueueControl ¶
type QueueControl interface {
PauseQueue(ctx context.Context, name string) error
ResumeQueue(ctx context.Context, name string) error
PausedQueues(ctx context.Context) ([]string, error)
}
QueueControl lets an operator stop a queue without stopping the servers. The usual reason is a downstream outage: the work is still worth doing, just not right now, and pausing beats letting ten thousand jobs burn their retries against a dead API.
Pausing affects only Fetch. Jobs continue to enqueue, nothing is lost, and resuming drains the backlog in the normal order.
type QueueStat ¶
type QueueStat struct {
Name string
Enqueued int64
Running int64
Paused bool
// OldestEnqueued is the age of the head of the queue, the number to alert on.
OldestEnqueued time.Duration
}
QueueStat is one row of the dashboard's queue table.
type Result ¶
type Result struct {
State State // Succeeded, Retrying, Dead or Cancelled
RetryAt time.Time // when State is Retrying
Err string
}
Result is the outcome a server reports for a job it holds the lease on.
type Schedule ¶
type Schedule struct {
ID string // caller-chosen, stable across deploys
Kind string
Queue string
Payload []byte
Cron string
Timezone string
LastRun time.Time
NextRun time.Time
Paused bool
}
Schedule is a recurring job definition. The next due time is computed by the server, not the driver, so drivers never need a cron parser.
type Schedules ¶
type Schedules interface {
SaveSchedule(ctx context.Context, s Schedule) error
ListSchedules(ctx context.Context) ([]Schedule, error)
RemoveSchedule(ctx context.Context, id string) error
// DueSchedules returns schedules with NextRun <= now and Paused false.
DueSchedules(ctx context.Context, now time.Time) ([]Schedule, error)
// AdvanceSchedule records that a run was fired. The caller holds the Locker
// lock for the schedule, so this needs no atomicity of its own.
AdvanceSchedule(ctx context.Context, id string, lastRun, nextRun time.Time) error
// SetSchedulePaused stops or resumes a schedule without deleting it, which is
// what an operator wants during an incident: the definition and its history
// survive, and resuming does not re-fire whatever was missed.
//
// Setting it on an absent schedule is not an error.
SetSchedulePaused(ctx context.Context, id string, paused bool) error
}
Schedules stores recurring job definitions. Cron expressions are parsed by the server, which writes back the computed NextRun, so drivers store opaque text.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server runs jobs. Its whole job is to turn the five Storage methods into a correct worker loop, which is the proof that the contract is sufficient: if the server needs something a driver cannot express, the contract is wrong.
func NewServer ¶
func NewServer(store Storage, cfg ServerConfig) *Server
NewServer returns a worker bound to store. Register handlers on it with each job type's Handle method before calling Run.
type ServerConfig ¶
type ServerConfig struct {
// Queues are polled in the order given, so the first is the highest priority.
Queues []string
// Concurrency is the number of jobs this server runs at once.
Concurrency int
// Lease is how long a claimed job stays owned without a heartbeat. It bounds
// how long a crashed server's work sits idle, so keep it well under the
// patience of whoever is waiting for the job.
Lease time.Duration
// PollInterval is how long to wait after an empty fetch.
PollInterval time.Duration
// ScheduleInterval is how often recurring jobs are checked. Cron resolution
// is one minute, so the default trades a little latency for far fewer
// queries; lowering it does not make a cron job fire more often.
ScheduleInterval time.Duration
// ScheduleMaxAttempts is MaxAttempts for jobs created from a schedule.
ScheduleMaxAttempts int
// Backoff returns how long to wait before retrying a job that failed on
// attempt n (1-based). Defaults to exponential with jitter, capped at an hour.
Backoff func(attempt int) time.Duration
Logger *slog.Logger
}
ServerConfig tunes a Server. Every field has a working default, so the zero value runs ten workers on the "default" queue.
type ServerInfo ¶
type ServerInfo struct {
ID string
Hostname string
Queues []string
Concurrency int
StartedAt time.Time
HeartbeatAt time.Time
}
ServerInfo is one running server, as reported by Heartbeat and listed by Monitor.Servers.
type State ¶
type State string
State is the lifecycle position of a job. The set is closed: drivers must not invent states, because the dashboard and the retry policy both switch on it.
Scheduled ──due──> Enqueued ──fetch──> Running ─┬─> Succeeded
^ ^ ├─> Retrying ──backoff──> Enqueued
└──────────────────┴── lease expiry ────────┼─> Dead
└─> Cancelled
type Storage ¶
type Storage interface {
// Enqueue persists jobs. A job whose ScheduledAt is in the future is stored
// in StateScheduled and becomes fetchable when it comes due; otherwise it is
// stored in StateEnqueued.
//
// A job whose ID already exists is ignored, silently. Callers rely on this
// for idempotency: the scheduler derives a job id from (schedule, tick), so a
// server crashing between enqueuing a tick and recording it refires the same
// tick and the duplicate is a no-op instead of a second run.
//
// Ignored means ignored — not overwritten. Overwriting would resurrect a
// finished job as a fresh one and run the work twice, which is the exact
// failure the rule exists to prevent.
//
// When UniqueKey is set and a non-terminal job already holds that key, the
// new job is dropped silently rather than returning an error.
Enqueue(ctx context.Context, jobs []*Job) error
// Fetch never returns a job from a paused queue, if the driver implements
// QueueControl. The check belongs here and not in the caller: a paused queue
// must be paused for every server, including one that has not noticed yet,
// and a server filtering its own list would keep draining a queue an operator
// just stopped.
//
// Fetch atomically claims at most n due jobs from queues, moving each to
// StateRunning with a lease held by owner until now+lease. An empty result is
// not an error.
//
// A job is due when its ScheduledAt has passed according to the STORAGE's
// clock, never the calling process's. That is the only clock every server
// shares, so it is the only one they can agree on. The cost is that a job
// enqueued with ScheduledAt = time.Now() on a host whose clock runs ahead of
// the storage is not instantly fetchable — it waits out the skew, then the
// next poll. Callers that need a job to run now should not depend on it being
// fetchable in the same millisecond it was enqueued.
//
// Jobs are returned in Priority then ScheduledAt order, best-effort within the
// driver's priority granularity. SQL drivers honour Priority exactly. Redis
// cannot: a sorted set score encodes one dimension, so ordering by priority
// and filtering by due time at once needs a structure per priority level, and
// an unbounded number of levels is unbounded by user input. Drivers with
// coarse priority clamp it and document the range.
Fetch(ctx context.Context, queues []string, n int, owner string, lease time.Duration) ([]*Job, error)
// Extend renews the lease on jobs still running under owner. Jobs no longer
// owned are skipped, not reported.
Extend(ctx context.Context, ids []string, owner string, lease time.Duration) error
// Finish applies r to a job owner holds. On Retrying the job returns to
// StateScheduled with ScheduledAt set to r.RetryAt and Attempt incremented.
//
// Any terminal state (Succeeded, Dead, Cancelled) must also clear UniqueKey.
// Otherwise a job keyed report:2026-10-05 holds that key forever once it
// succeeds, and can never be enqueued again — including by a manual retry
// from the dashboard.
Finish(ctx context.Context, id, owner string, r Result) error
// Reclaim returns jobs whose lease expired to StateEnqueued and reports how
// many were recovered. Every server calls it periodically, so it must be
// safe to run concurrently on every node.
Reclaim(ctx context.Context) (int, error)
Close() error
}
Storage is everything a driver MUST implement. Five methods: that is the whole bar for running one-off, delayed and retried jobs.
Deliberately absent: sets, hashes, lists, counters. Hangfire's storage API leaks Redis data structures into every driver, which is why its SQL storages are a pile of emulation tables. This contract speaks jobs, queues and leases, and each driver answers it however its engine is good at.
Delivery is at-least-once. Every implementation must satisfy:
- Fetch is atomic across processes. Two servers polling the same queue never receive the same job. SQL drivers use SELECT ... FOR UPDATE SKIP LOCKED; Redis uses an atomic pop into a lease set.
- A job in StateRunning whose lease has expired is recoverable by Reclaim, even if the server that held it never comes back.
- Finish is ignored (not an error) when owner no longer holds the lease. The job was already reclaimed and may be running elsewhere; the late result must not overwrite the new attempt.
Everything beyond this is an optional interface below, discovered by type assertion. A driver that implements only Storage still runs jobs; it just lights up fewer panels in the dashboard.
type Transition ¶
Transition is one entry in a job's state history, shown on the job detail page.
type Tx ¶
type Tx interface {
ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
}
Tx is the subset of a database handle needed to insert a job inside a caller's transaction. *sql.Tx satisfies it. Native pgx users wrap their pool with pgx/stdlib, or use the pgx driver's own EnqueueTxPgx.
type TxEnqueuer ¶
TxEnqueuer is implemented by SQL drivers. It makes a job and the rows that justify it commit together, so a job can never reference a row that rolled back. Redis cannot offer this; code that needs it should type-assert and fail loudly at startup rather than degrade silently.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
driver
|
|
|
memory
Package memory is an in-process jobsync storage driver.
|
Package memory is an in-process jobsync storage driver. |
|
mysql
Package mysql is the MySQL/MariaDB jobsync storage driver.
|
Package mysql is the MySQL/MariaDB jobsync storage driver. |
|
postgres
Package postgres is the reference jobsync storage driver.
|
Package postgres is the reference jobsync storage driver. |
|
redis
Package redis is the Redis jobsync storage driver.
|
Package redis is the Redis jobsync storage driver. |
|
examples
|
|
|
dashboard
command
Command dashboard is a runnable jobsync example: a worker producing traffic, with the dashboard mounted over it.
|
Command dashboard is a runnable jobsync example: a worker producing traffic, with the dashboard mounted over it. |
|
Package storagetest is the conformance suite for jobsync Storage drivers.
|
Package storagetest is the conformance suite for jobsync Storage drivers. |




