natslink

package module
v1.8.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 15, 2026 License: MIT Imports: 15 Imported by: 0

README

CI Go Reference Go Report Card

Self-supervising NATS connections for Go: one connection, one subject, zero babysitting.

natslink wraps nats.go in small units that never give up. Each unit owns a single connection bound to a single subject and takes care of dialing, exponential-backoff reconnects, rebuilding the connection after a permanent close, re-subscribing, bounded back-pressure, and periodic status reporting. You write the handler; the library keeps the pipe alive.

  • Subscriber - one connection subscribing to one subject. Three dispatch modes (bounded worker pool, synchronous, goroutine-per-message), an optional Gate to drop messages at the door, and Pause / Resume to withdraw interest without dropping the connection.
  • Publisher - one connection publishing to one subject. Same supervision; a failed publish returns an error for the caller to handle.
  • PublisherPool - N publishers with round-robin and health-first failover. Traffic flows through the healthy members while one is reconnecting. Mirrored pools run one independent send line per endpoint, so a stalled endpoint never delays the others.
  • Mirroring - NewMirroredPublisher / NewMirroredPublisherPool / NewMirroredSubscriber fan the same subject out to M independent NATS endpoints. Publishes go to every endpoint; subscribers receive every copy, or exactly one when you opt in to the built-in DedupSet (byte-identical copies collapse without decoding, first arrival wins). The return types are the plain *PublisherPool / *Subscriber, so the rest of your code does not change.

Only two dependencies: nats.go and the standard library.

Install

go get github.com/Jolly23/natslink@latest

Requires Go 1.25 or newer.

Mental model (read this first)

  1. One connection = one subject = one self-supervising unit. Need more subjects? Create more units. Failures stay isolated.

  2. Failures fall into four classes, each with one owner:

    Phase Class Surface What you do
    Startup Environment (bad address, no route, bad credentials) Probe returns an error MustProbe(url) - panic, fail fast
    Startup Configuration (malformed URL or subject) New* / Start return an error MustStart() - panic, fail fast
    Runtime Ordinary disconnect (server restart, flaky network, stalled peer) automatic reconnect nothing; it never gives up
    Runtime Permanent close (fatal protocol error from the server) automatic rebuild nothing; it never gives up

    Startup errors must crash you. Runtime failures must never bother you. A successful start is a promise: everything is ready, and the unit will never stop trying.

  3. Instances are single-use. After Stop() create a new one.

  4. Supervision parameters are not configurable. Dial timeout, reconnect and rebuild backoff, heartbeat probing and write deadlines are package constants tuned in production (see the top of natslink.go). Exposing them invites every project to tune its own broken set.

  5. Back-pressure is always visible. When a handler cannot keep up, messages are dropped in a bounded way and counted (Dropped, SlowConsumers); nothing queues without limit. The one exception is the explicit Workers: GoPerMessage subscriber mode (unbounded concurrency, no back-pressure signal), which you choose only when you know the traffic and must not lose messages.

Five-minute start

import (
    "log/slog"

    "github.com/Jolly23/natslink"
    "github.com/nats-io/nats.go"
)

url := "nats://token@127.0.0.1:4222"

// 1. Fail fast on environment errors (<= 5s). One probe per URL per process.
natslink.MustProbe(url)

// 2. Subscribe: one connection, one subject.
sub, err := natslink.NewSubscriber(natslink.SubscriberOptions{
    Options: natslink.Options{
        URL:    url,
        Topic:  "events.tx",
        Name:   "mynode-sub-tx",   // default natslink-sub-<topic>; shows up in server monitoring and logs
        Logger: slog.Default(),    // anything with Info/Warn/Error(msg string, kv ...any)
    },
    Handler: func(m *nats.Msg) { /* called concurrently by default, see "Subscribing" */ },
})
if err != nil {
    panic(err) // configuration error (bad subject, ...), fail fast
}
sub.MustStart()
defer sub.Stop() // blocks until in-flight callbacks return

// 3. Publish through a pool of N connections with health-first failover.
pool, err := natslink.NewPublisherPool(4, natslink.PublisherOptions{
    Options:    natslink.Options{URL: url, Topic: "events.block", Name: "mynode-pub-block"},
    MaxPayload: 16 << 20, // application-level size cap, rejected with ErrOversized; 0 = unlimited
})
if err != nil {
    panic(err)
}
pool.MustStart()
defer pool.Stop()

if err := pool.Publish(payload); err != nil {
    // only when every member failed (already counted in pool.Dropped())
}

Start dials an instance's connections (a mirrored subscriber's endpoints, a pool's members) with at most eight in flight, so startup takes the slowest batch rather than the sum, and a subscriber's workers are running before the first endpoint can deliver. Stop may be called at any point, including during that initial dial: it never leaves a connection behind, and the interrupted Start returns ErrStopped.

The three-line startup discipline: MustProbe (environment) → New* (configuration) → MustStart. Note that MustStart succeeding does not mean the connection is up (see the failure model); the fail-fast guarantee comes from MustProbe.

Failure model

At startup, MustProbe is the only correct check. Start / MustStart enable RetryOnFailedConnect, so an unreachable server or a bad token is not a Start error: the connection is created and retried in the background forever. That is exactly what you want at runtime, but it means MustStart cannot catch environment mistakes. So begin your process with:

natslink.MustProbe(url) // one throw-away connection checks reachability and auth; panics within 5s (credentials redacted)

One probe per URL; afterwards start as many connections as you like. Do not use "WaitConnected + panic on every connection" as a startup check: with many connections each one waits out its own timeout, and a bad token also waits the full timeout because auth errors are deliberately treated as recoverable (see below). WaitConnected(timeout) stays available as a readiness helper for tests or for "must be online before the first message" situations.

At runtime there are two layers, both retrying forever:

  1. Ordinary disconnect (server restart, network blip, stalled peer caught by heartbeats) → state RECONNECTING, exponential backoff from 500 ms doubling to a 5 s cap with +50 % jitter. Subscriptions are restored automatically and the publisher's 16 MB reconnect buffer is flushed. A server restart of ten or twenty seconds is fully absorbed here.
  2. Permanent close (a fatal protocol error puts nats.go into CLOSED, which layer 1 cannot recover from) → the whole connection is rebuilt: redial, re-subscribe, atomically swap in. Backoff from 2 s doubling to a 30 s cap.

How fast a dead connection is noticed: clean breaks (restart, kill, RST) are seen by the read loop within milliseconds. A silently dead connection (NAT or firewall dropped the mapping) is caught by a 3 s heartbeat with two consecutive misses, so about 9 s worst case (nats.go defaults take ~6 minutes). A single late PONG never kills a healthy connection.

Three defences against "the client silently gave up after a server restart":

  1. nats.go stops after MaxReconnects=60 by default (roughly two minutes of outage) → MaxReconnects(-1), retry forever.
  2. During a server's warm-up it may answer with auth errors, and nats.go gives up permanently after the same auth error twice → IgnoreAuthErrorAbort() (covered by an integration test).
  3. A fatal protocol error (an unknown -ERR) goes straight to CLOSED and neither of the above helps → the ClosedHandler triggers a full rebuild.

Even an unknown failure mode is never silent: a status line is logged every minute (ReportInterval; negative disables it) and a degraded warning is added while disconnected.

Subscribing

Dispatch mode is chosen with SubscriberOptions.Workers:

Setting Behaviour Use when
default (0) 8 worker goroutines + bounded queue (QueueSize, default 4096) The normal case. Concurrency is capped; when the queue is full new messages are dropped and counted (Dropped) instead of growing goroutines or memory.
Workers: N N worker goroutines Tune per subject and handler cost. Workers: 1 gives strict ordering without blocking delivery.
Workers: natslink.SyncMode Handler runs on the nats.go delivery goroutine Lowest latency (no queue hand-off). The handler must not block: locks, I/O or millisecond work belong in a worker pool. Backlog is bounded by the client pending buffer (default 512k msgs / 64 MB); overflow is counted in SlowConsumers.
Workers: natslink.GoPerMessage One new goroutine per message, unbounded concurrency Mixed traffic where most messages are light, a few carry heavy synchronous work, and none may be dropped. A bounded pool would let the heavy ones occupy every worker and head-of-line block the light ones, then drop during bursts; when the real throughput cap is a downstream lock, extra goroutines merely queue on that lock. Cost: no back-pressure signal at all (Dropped is always 0, SlowConsumers almost never fires); goroutine count floats with load. Only pick this when you understand both the producer rate and the consumer capacity.

Handler concurrency contract (read before writing a handler):

  • Worker-pool mode calls the handler from up to Workers goroutines concurrently, and processing order is not guaranteed to match publish order (workers compete for one queue). Guard shared state yourself; use Workers: 1 or SyncMode when strict ordering matters.
  • Sync mode is serial and ordered, but blocks the delivery goroutine.
  • Goroutine-per-message mode calls the handler with unbounded concurrency and no ordering, and Stop() does not wait for those callbacks (unlike the worker-pool guarantee). Release handler dependencies only after you have confirmed the callbacks are done.

QueueGroup: unset means broadcast, every subscriber receives everything. Set it and members of the same group compete: each message goes to exactly one of them. Multiple replicas that each need the full stream must leave it unset; replicas that share the load must set it. Getting this wrong in either direction produces no error, only wrong behaviour.

Gate (v1.4.0): an optional func() bool evaluated at the delivery entry point for every message. true delivers to the handler; false drops the message on the spot and counts it in Stats.Gated. nil always passes. The check happens before enqueueing or spawning in all three modes. The gate runs on the nats.go delivery goroutine (M of them concurrently under mirroring), so it must be concurrency-safe and very cheap (an atomic load); a slow gate blocks delivery and triggers SlowConsumers. Typical use: hold all processing while the host is not ready yet, then let it flow without rebuilding the subscription. In worker-pool mode the gate is evaluated at enqueue time, so a message already queued when the gate closes is still delivered; re-check inside the handler if you need exact timing.

Pause / Resume (v1.5.0): withdraw and restore the subscription at runtime without dropping the connection. Pause() sends UNSUB on every endpoint's current connection: the server stops delivering, and interest is withdrawn hop by hop across leaf nodes and routes, so upstream servers stop forwarding that subject to you at all (real bandwidth saved; a Gate drops at the door but the bytes still arrive). Connection, heartbeats, reconnect and rebuild all keep running; Stats.Connected is unchanged, Stats.Paused is true and Stats.Pauses increments. Resume() re-SUBs on each endpoint's most recent successfully set-up connection; messages return to the same handler and worker pool. Both are idempotent and return ErrStopped after Stop. A rebuild during a pause swaps in a new connection without subscribing; nats.go's own reconnect only replays subscriptions still on record, so an UNSUBed one never comes back by accident. Neither path needs action from the caller. Edge: messages already in flight before UNSUB, or already queued in the worker pool, are still delivered (same window as Gate). Unsubscribe / Subscribe take the underlying connection lock and can block up to the write deadline (5 s) against a stalled peer, so do not call them synchronously on a latency-critical path.

Partial-failure self-healing (v1.5.1): if Resume fails to SUB on some endpoint (or Pause fails to UNSUB), the paused state still flips, the errors are returned joined, and a single in-package healer goroutine retries the failed endpoints with the rebuild backoff (2 s doubling to 30 s), aligning each round to the paused state at that moment, and exits once everything matches or Stop is called. The caller never retries. Logs: [natslink] subscribe retried after failed resume / unsubscribe retried after failed pause on success and subscription reconcile failed, retrying (Warn) per failed endpoint per round; Stats.Diverged > 0 while unconverged. The healer holds the internal subscription lock across SUB / UNSUB like Pause / Resume and rebuild do, so against a stalled peer each endpoint can block up to 5 s (N endpoints → N × 5 s) during which the host's Pause / Resume, rebuild and Stats() queue behind it. With nats.go 1.53 a SUB / UNSUB on a live connection only fails while the connection is draining (after which it closes and rebuild takes over), so this is a correctness backstop rather than a hot path.

Deduplication (v1.7.0): Dedup: natslink.NewDedupSet(entries, window) drops a message whose payload is byte-identical to one recorded recently, before it is queued or dispatched (Stats.Deduped). It is a general byte-level filter, not a mirroring feature: hang it on one subscriber, share one set across several (identical payloads on different subjects then collapse too, say one event published on three feeds), give every subscriber a process owns the same set, or call set.Seen(data) yourself on a path that does not go through a Subscriber. The typical case is a mirrored subscription, where every message arrives once per endpoint: the first copy wins and the others never reach the handler, the queue or a goroutine. The set remembers the last entries distinct payloads (0 = 4096) as 64-bit hash/maphash values in a ring buffer plus a presized map, so a check is one hash outside the lock and one map lookup inside it, with no allocation after warm-up; a hit does not refresh the entry, so flooding one message cannot pin it. The window is a capacity, optionally bounded by age (window > 0 makes an entry older than that stop counting, so a legitimately re-sent identical message on a quiet subject is not swallowed minutes later; 0 disables age expiry). Memory is about 40 bytes per entry. One set can be shared by several subscribers, in which case identical payloads on different subjects collapse too. Empty payloads bypass the set. DedupKey func(*nats.Msg) (uint64, bool) replaces the payload hash with a key you compute (for example a hash already present at a fixed offset, sparing the hash of a multi-megabyte message); ok=false delivers without consulting the set. What the set does not do: collapse messages that carry the same content in different bytes (a sender field, a timestamp), so content-level deduplication (sequence numbers, a hash carried in the payload) stays in the application; and it is a latency and CPU saving, not a delivery guarantee: a copy arriving after the set has forgotten the first one is delivered again. Hashing cost is the only per-message cost: about 50 ns for 64 B, 0.3 µs for 4 KB and 130 µs for 2 MB on one core.

Server-side rejection (v1.8.2): a SUB the server refuses (a permissions violation on the subject) is answered with -ERR after the local Subscribe already succeeded, and nats.go never retries it. The error reaches the library through nats.go's async callback, which only triggers a probe of the endpoint's current state (a temporary subscription on the same subject, one flush round trip, LastError before and after): the server's answer now decides, so a late or stale callback can never invalidate a healthy subscription. A rejected probe drops the handle (Stats.Diverged goes to 1 for that endpoint) and the SUB is re-sent on the rebuild backoff schedule (2 s doubling to 30 s, seeded by the number of consecutive rejections), so an ACL repaired, or revoked, by a config reload takes effect without any action from you. The server's subscription limit is not auto-recovered: it is logged (async error) for the operator. Log line [natslink] subscription rejected by server, retrying.

Subjects pass straight through to NATS, so subscriptions may use wildcards (* / >, with > only as the last token; a misplaced > is rejected at construction because the server would reject the subscription on every reconnect); one connection still binds exactly one subject expression. Client buffer limits (PendingMsgLimit / PendingBytesLimit) are documented on SubscriberOptions and rarely need changing in worker-pool mode.

Publishing

pub, err := natslink.NewPublisher(natslink.PublisherOptions{
    Options:    natslink.Options{URL: url, Topic: "events.tx", Name: "mynode-pub"},
    MaxPayload: 4 * 1024,
})
if err != nil {
    panic(err)
}
pub.MustStart()
defer pub.Stop()

Publish semantics (safe for concurrent use):

  • nil = the message was accepted by the connection, not delivered. When connected it is written immediately; while RECONNECTING it goes into the 16 MB reconnect buffer and is flushed after reconnect.
  • Replay is best effort (core NATS has no delivery guarantee). If the connection goes permanently CLOSED while buffering, the old buffer is lost with it; a replay may also arrive before the peer has re-subscribed. Build acknowledgements at the application layer if you need delivery guarantees.
  • error = the message was not accepted (counted in Stats.Dropped; mirrored pools add ErrQueueFull for "no endpoint queue could take it"): ErrStopped (instance used up) / ErrNotRunning (not started) / ErrOversized (over MaxPayload) / ErrNoConn (rebuild window, match with errors.Is to fail over) / any underlying error such as nats.ErrReconnectBufExceeded.
  • data is copied before Publish returns; reuse your buffer immediately.
  • Stop() tries to drain the buffer (1 s timeout). While disconnected the drain necessarily fails and buffered messages are dropped (Warn log with the byte count).
  • Endpoint pinning (v1.3.2): connections ignore cluster gossip (IgnoreDiscoveredServers). The server pool contains only the URLs you configured, so when your endpoint goes down the client keeps reconnecting to that endpoint and never drifts to another cluster member (a production client once wandered off to a remote node and never came back). This is a deliberate regional lock; for multiple endpoints use mirroring or an explicit comma-separated list.
  • Bounded shutdown (v1.3.1): Stop() blocks at most stopGrace (5 s) per connection. Drain and close run on a side goroutine, so a stalled peer (zero window or half-open socket holding the write path's connection lock) cannot hang your shutdown; the close continues in the background. PublisherPool.Stop and mirrored Subscriber.Stop stop members in parallel, so pool shutdown costs max, not sum. The normal path stays in the millisecond range. Background: a production shutdown once hung until systemd sent SIGKILL because nats.go's FlushTimeout(1s) only bounds the wait for PONG, not the wait for the connection lock, and the lock was held by a write blocked on a 60 s deadline. The same release cut the per-write socket deadline to 5 s (flusherTimeout, nats.go default 60 s), which brings the worst-case recovery from "a wedged write blinds the heartbeat" down from minutes to roughly 10-30 s under write pressure (an idle connection stays at ~9 s). While wedged, the flusher discards one aggregated batch of already accepted messages every ~5 s (their Publish returned nil); this only shows up in async error logs, not in Stats.Dropped, so pool-level Dropped underestimates loss in that window.

Large-message checklist (go through it before sending anything over 1 MB):

  1. The server's max_payload must be ≥ your MaxPayload. The NATS server default is 1 MB; oversized messages are rejected or the connection is dropped. Probe checks reachability and auth only, not payload limits, so this mistake looks like a green startup followed by every publish failing.
  2. The reconnect buffer is 16 MB: during an outage two 8 MB messages fit, the third gets ErrReconnectBufExceeded. Do not rely on the buffer for large messages over long outages. The 5 s write deadline also puts a floor on replay bandwidth: the pending backlog (up to 16 MB) is written as a single Write under a single 5 s deadline, so the link must sustain backlog/5 s (16 MB → ~3.2 MB/s ≈ 26 Mbit/s) or the whole batch is silently dropped and the client reconnects again. Fine for a local or same-region server; publishers on constrained cross-region links must either accept best-effort replay or size the backlog to "5 s × link throughput".

Publisher pool (multiple connections + failover)

pool, err := natslink.NewPublisherPool(6, natslink.PublisherOptions{
    Options:    natslink.Options{URL: url, Topic: "events.tx", Name: "ingest-tx"},
    MaxPayload: 4 * 1024,
})
if err != nil {
    panic(err)
}
pool.MustStart()
defer pool.Stop()

_ = pool.Publish(payload)                 // routing semantics below
_ = pool.WaitConnected(10 * time.Second)  // wait for all N members (optional; tests / readiness gates)
n := pool.ConnectedCount()                // healthy members right now
d := pool.Dropped()                       // real pool-level loss (see "Statistics")

Routing (latency first): starting at the round-robin cursor, Publish makes two passes. The first pass tries only members that are currently connected, so a message leaves on a healthy connection immediately instead of sitting in a disconnected member's reconnect buffer. Only when no member is healthy does the second pass fall back to "accepted into the reconnect buffer counts as success". Any member accepting returns nil; all failing returns an error and increments the pool-level Dropped().

Mirroring (active-active across independent NATS endpoints)

NewMirroredPublisher / NewMirroredPublisherPool / NewMirroredSubscriber mirror one subject across M mutually independent NATS endpoints: each publish goes to every endpoint, each subscriber subscribes on every endpoint and the handler receives every copy (or exactly one with a DedupSet, see "Subscribing"). Mirroring is only a constructor. The returned types are the plain *PublisherPool / *Subscriber; every method behaves the same and the host code changes on exactly one line.

Typical setup: one machine runs two local leaf nodes that connect upstream to different regions (two trees that never meet). Publishing races both entry points; subscribing takes the minimum of both paths. The library collapses byte-identical copies when asked to; content-level deduplication stays with the consumer (monotonic sequence numbers, a hash carried in the payload), because two publishers describing the same event in different bytes are not duplicates to the library.

NewMirroredPublisher(urls, opts) is the single-connection-per-endpoint form (at most 5 endpoints; a publish is sent once per endpoint, so the fan-out is deliberately capped). NewMirroredPublisherPool(urls, n, opts) gives each endpoint n connections when one is not enough. With exactly one URL every mirrored constructor degrades to its plain counterpart: NewMirroredPublisher becomes NewPublisherPool(1, opts), NewMirroredPublisherPool becomes NewPublisherPool(n, opts) and NewMirroredSubscriber becomes NewSubscriber (same naming, no -e0 suffix, no send queue, synchronous Publish), so a deployment can grow from one tree to several without a code change.

urls := []string{"nats://tk@127.0.0.1:4222", "nats://tk@127.0.0.1:4223"}
natslink.MustProbeEach(urls...) // fail fast per endpoint: M endpoints are M promises

pub, err := natslink.NewMirroredPublisher(urls, natslink.PublisherOptions{
    Options: natslink.Options{Topic: "events.tx", Name: "edge-pub-tx", Logger: slog.Default()},
})
if err != nil {
    panic(err)
}
pub.MustStart()
defer pub.Stop()
_ = pub.Publish(payload) // sent once per endpoint; nil if any endpoint accepted

sub, err := natslink.NewMirroredSubscriber(urls, natslink.SubscriberOptions{
    Options: natslink.Options{Topic: "events.block", Name: "edge-sub-block", Logger: slog.Default()},
    Handler: onBlock,                              // called once per block: the first copy wins
    Dedup:   natslink.NewDedupSet(4096, 2*time.Second), // byte-identical copies from the other endpoint are dropped
})

Semantics at a glance:

  • Not the same as a comma-separated URL. A comma list in Options.URL is ordered failover within one endpoint: only one server is connected at a time, in the order written (natslink disables nats.go's default server-pool shuffle), and the next one is tried after a disconnect. nats.go stays on whichever server it lands on and never fails back, so this is not full active/standby; the first entry is merely the preferred one for the initial dial and each reconnect round. The mirror urls are active-active across endpoints: all connected, all used. The two compose: each element of urls may itself be a comma list ("ordered failover inside an endpoint, mirroring across endpoints"). MustProbeEach on a comma-list endpoint can take up to 5 s per listed server, serially.
  • Every endpoint is an independent send line. A mirrored pool gives each endpoint its own bounded queue (4096 messages, 16 MB) and its own goroutine. Publish copies the message once, hands that copy to every queue without blocking and returns nil as soon as any queue accepts; the endpoint goroutines then run the two-pass routing on their own. A stalled endpoint (a wedged write holding its connection lock, a saturated link) backs up only its own queue and, once that is full, drops only its own copies, while the caller and the other endpoints never wait for it. Mirroring is redundancy: one endpoint failing just means one fewer path, not a loss. Only when no endpoint queue can take the message is ErrQueueFull returned, and only a message that every endpoint failed to send counts in the pool-level Dropped() (settled asynchronously). Admission reserves queue bytes per endpoint before copying the payload, so a publish that nothing can admit is rejected for a few nanoseconds without copying anything. Send failures on one endpoint show up in that member's Stats.Dropped, a backlog in its QueueLen, and copies its queue discarded (full, or left over when Stop gave up on a stalled send) in its QueueDropped (queue_dropped= in the status line). Stop lets every send goroutine flush its queue first, waits for them in parallel for at most one stop grace, and a concurrent second Stop waits for that same sequence. A single-endpoint pool keeps the synchronous Publish of the plain pool.
  • Subscribers share one worker pool and one set of counters. Workers is the total concurrency cap across endpoints and QueueSize is one shared queue. Ordering guarantees from Workers: 1 / SyncMode hold only within one endpoint; copies from different endpoints interleave. QueueGroup applies per endpoint.
  • Observing endpoint degradation: connection names carry an -e<i> suffix so server-side monitoring can attribute them. Pool Stats() flattens all members, so one endpoint being down shows as a block of members with Connected=false. A mirrored subscriber's merged Stats().Connected uses an all-endpoints threshold: any endpoint down → false, so your existing "degraded" alert covers it with no extra monitoring.
  • Mirroring × SyncMode warning: sync mode's "serial and ordered" promise does not survive mirroring. M endpoints have M delivery goroutines, so the handler is called concurrently. A handler relying on sync mode to avoid locks is a data race under mirroring (there is a test proving it).
  • Mirrored Dropped counts copies unless you deduplicate. Without a DedupSet each message occupies M queue slots, so under overload Dropped is inflated by roughly M and content is lost only when all copies of a message are dropped; divide by M before judging. With a DedupSet only the first copy reaches the queue, Deduped grows by M-1 per message in steady state, and Dropped means what it means for a single connection.
  • Each endpoint heals independently and knows nothing of the others. One endpoint down for a long time = the race degrades to a single path (no loss, half the benefit).
  • Topology discipline (learned the hard way): mirroring only means something when the endpoints really are two independent trees. Two local leaf nodes must have different server_names; with the same name the upstream treats them as the same origin cluster and deduplicates or loop-protects, collapsing your two copies into one or black-holing both. Check max_payload hop by hop on each endpoint's path as well.

Statistics

Stats() can be called concurrently at any time. Counters accumulate across rebuilds (a brief dip may be observed at the instant of a rebuild swap; clamp negative deltas to zero in rate monitors).

Field Meaning
Name Connection name (pool members get -<i>, mirror endpoints -e<i>): the attribution key for a degraded connection
Status / Connected / Server / Cluster Current connection state
Reconnects Successful automatic reconnects
Rebuilds Full connection rebuilds after a permanent close
InMsgs / OutMsgs / InBytes / OutBytes Traffic counters
Dropped pub: failed publishes; sub: worker-queue overflow drops (worker-pool mode only; always 0 in sync and goroutine-per-message modes)
Gated sub: messages dropped by the Gate at the delivery entry point (all modes; 0 without a gate; counts copies under mirroring)
Deduped sub: copies dropped by the DedupSet after the gate and before the queue (all modes; 0 without a set; M-1 per message in a healthy M-endpoint mirror)
Paused / Pauses sub: whether currently paused (connection up, subscription withdrawn) / number of successful pause transitions (idempotent repeats not counted). pub: always false / 0
Diverged sub: endpoints whose connection is alive but whose subscription state disagrees with the paused state (v1.5.1). > 0 = the healer is retrying; persistently > 0 → look for subscription reconcile failed. Endpoints inside a rebuild window are not counted. pub: always 0
SlowConsumers Number of slow-consumer events in which the client dropped messages (the only loss signal in sync mode; events, not messages, so do not add to Dropped)
QueueLen / QueueCap sub: worker-queue level (0 in sync and goroutine-per-message modes); pub: in a mirrored pool, the send-queue level of the member's endpoint (0 for plain publishers)
QueueDropped pub, mirrored pool member only: copies its endpoint's send queue discarded (queue full by entries or bytes, or still queued when Stop gave up on a stalled send goroutine); send failures are counted in Dropped instead

Wiring monitoring: subscriber loss = Dropped (worker pool) plus SlowConsumers (sync); goroutine-per-message subscribers do not drop and have no back-pressure signal, so watch process-level goroutine count and RSS instead. Publisher-pool loss is pool.Dropped() only: a single member's Stats.Dropped includes failover intermediate failures (the message may have left on another member), so summing members produces false alarms.

Logging

Every connection logs with the [natslink] prefix through the Logger you pass in Options (any type with Info / Warn / Error(msg string, kv ...any); *slog.Logger and go-ethereum's log.Logger both qualify, the default is slog.Default()). Events: starting (with version), connected, reconnected, disconnected, permanently closed, rebuilding, connection rebuilt, periodic status, and degraded while disconnected, plus the Pause / Resume healer lines listed above. Errors never contain credentials: the configured URL's userinfo is redacted everywhere it could leak, including URLs re-rendered by net/url on parse failures and their %q-escaped and percent-encoded forms. Errors keep errors.Is classification but do not expose an Unwrap chain to the raw sensitive error; errors you obtain through the Conn() escape hatch are your own responsibility.

Notes

  • Instances are single-use: after Stop() create a new one (further calls return ErrStopped). Pause / Resume are not Stop: they only withdraw or restore the subscription while the connection stays up.
  • Subscriber.Stop() blocks until in-flight callbacks return, so handler dependencies may be released afterwards; a handler that blocks forever hangs Stop. Queued messages not yet started are discarded. This wait applies to worker-pool mode only (SyncMode / GoPerMessage callbacks are not awaited).
  • Publish, pool.Publish and Stats are safe for concurrent use.
  • ExtraNatsOptions is the escape hatch for policy options (TLS, custom dialer). Event handlers (rebuild, slow-consumer observation) are installed after your options and cannot be overridden. Overriding MaxReconnects, RetryOnFailedConnect and friends removes the corresponding defence; you own the consequences.
  • Conn() returns the underlying *nats.Conn for advanced use such as Request-Reply. It is outside the supervision contract and may be nil during a rebuild window.
  • Each Subscriber has its own worker pool; with many subjects, lower Workers accordingly.

Testing

Unit tests run without any server:

go test -short -race ./...

The integration suite needs two local NATS servers and the docker CLI (the restart and pause drills operate on the container that publishes the test port). make itest starts them, runs everything and tears them down; see CONTRIBUTING.md for the manual commands and the environment variables (NATS_TEST_URL, NATS_TEST_URL2, NATS_TEST_CONTAINER).

Versioning

Releases follow semantic versioning with git tags (v1.6.0). The natslink.Version constant mirrors the tag and is logged by every connection at start ([natslink] starting version=...), so operators can see which version each process runs. See CHANGELOG.md for the history and CONTRIBUTING.md for the release procedure.

License

MIT

Documentation

Overview

Package natslink packages "one NATS connection bound to one topic" as a self-supervising unit that works out of the box.

The goal is to let many programs share one battle-tested set of connection supervision logic:

  • Probe / MustProbe: fail-fast probing at startup. Environment errors (address / network / token) surface within about 5s, on the first line of the process's life; a URL only needs to be probed once.
  • Subscriber: one connection subscribing to one topic. Built-in connection supervision, automatic reconnect on disconnect, self-healing rebuild after a "permanent close" (with automatic re-subscription), and status / data statistics accumulated across rebuilds. Callbacks are dispatched in one of three modes: a bounded worker pool (default; back-pressure drops are visible), synchronous (SyncMode), or go-per-message (GoPerMessage, unbounded concurrency); see SubscriberOptions.Workers.
  • Publisher: one connection publishing to one topic. The same supervision and self-healing rebuild; a failed publish returns an error and the caller decides whether to fail over or count a drop.
  • PublisherPool: N Publishers with round-robin plus health-first failover (latency first: disconnected members are demoted to last-resort candidates, so a message always leaves on a healthy connection first).
  • Mirrored constructors NewMirroredPublisherPool / NewMirroredSubscriber: mirror the same topic onto M mutually independent NATS endpoints (publish once per endpoint, subscribe once per endpoint; deduplication happens on the consumer side). They return the ordinary types and are used exactly the same way; intended for active-active topologies such as "two local leaf nodes, each with its own upstream". See mirror.go and the README section on mirroring.

The standard startup sequence is MustProbe(url) (environment fail-fast) -> New* (configuration fail-fast) -> MustStart. After that, every runtime disconnect or permanent close is handled by the package's self-healing logic: it never gives up and never fails silently (see the README's mental-model and failure-handling sections).

The package depends only on nats.go and the standard library; it carries no application code.

Instances are single-use: after Stop they cannot be reused; create a new one.

Index

Constants

View Source
const GoPerMessage = -2

GoPerMessage, assigned to SubscriberOptions.Workers, selects go-per-message delivery (see that field).

View Source
const SyncMode = -1

SyncMode, assigned to SubscriberOptions.Workers, selects synchronous delivery (see that field).

View Source
const Version = "1.8.3"

Version is the package version. It mirrors the git tag (vX.Y.Z) and is logged at connection start, so operators can see which version each process runs.

Variables

View Source
var (
	ErrAlreadyStarted = errors.New("natslink: already started")
	ErrStopped        = errors.New("natslink: already stopped; create a new instance")
	ErrNotRunning     = errors.New("natslink: not running")
	ErrNoConn         = errors.New("natslink: no usable connection (rebuilding)")
	ErrOversized      = errors.New("natslink: payload exceeds MaxPayload")
	// ErrQueueFull is returned by a mirrored PublisherPool when no endpoint's
	// send queue could take the message (every endpoint is stalled or
	// backlogged); it is counted in the pool-level Dropped.
	ErrQueueFull = errors.New("natslink: every endpoint send queue is full")
)

Functions

func MustProbe

func MustProbe(natsURL string)

MustProbe is like Probe but panics on failure. Intended for process startup.

func MustProbeEach

func MustProbeEach(natsURLs ...string)

MustProbeEach calls MustProbe on every URL. It is the standard startup fail-fast for the mirrored constructors (multiple endpoints): any endpoint with an environment error panics immediately. Configuring M endpoints means making M promises. When an element is a comma-separated list (ordered failover within the endpoint), the servers are tried in the written order and reaching any one of them counts as success (matching that endpoint's runtime semantics); note that the worst case is then 5s per server, accumulated serially, so the "surfaces within 5s" promise is per server.

func Probe

func Probe(natsURL string) error

Probe opens a one-off connection to verify that natsURL is reachable and that authentication succeeds, then closes it immediately.

Why it exists: Start / MustStart enable RetryOnFailedConnect, so an unreachable server or a bad token is not a Start error (the connection is created anyway and retries forever in the background). That is exactly the self-healing behaviour wanted at runtime, but it silently swallows environment errors (typo in the address, firewall, expired token, services deployed in the wrong order) until someone reads the logs. The right semantics at startup are fail-fast: configuring NATS is a promise, and not being able to connect at startup is an environment error that should kill the process so the deployer notices right away.

Standard startup sequence (when several connections share one URL, probe it once; environment errors surface within about 5s instead of being stretched by N connections each waiting on their own):

natslink.MustProbe(url) // startup: environment error panics immediately, fail-fast
pool.MustStart()        // from here on an error can only be a configuration error (also fail-fast)
sub.MustStart()         // runtime disconnects are handled by the package's self-healing; it never gives up

Credentials in the URL are redacted in the returned error, so it is safe to log.

Types

type DedupSet added in v1.7.0

type DedupSet struct {
	// contains filtered or unexported fields
}

DedupSet remembers the hashes of the last N distinct payloads and reports whether a payload is a byte-identical copy of one of them. It is a general building block: give it to any Subscriber (SubscriberOptions.Dedup), share one across several or all subscribers of a process so copies arriving on different subjects collapse too, or call Seen directly on any path. The typical use is collapsing the M copies a mirrored subscription receives (one per endpoint) into a single callback without decoding anything.

It is a FIFO set of 64-bit maphash values: a ring buffer fixes the eviction order and a presized map does the lookup, so after warm-up a call allocates nothing. The hash is computed outside the lock; the critical section is one map lookup and, on a miss, one insert. A hit does not refresh the entry, so a flood of one message cannot pin it. With an optional time window an entry older than the window no longer counts as seen (useful on quiet subjects where a legitimately re-sent identical message would otherwise be dropped minutes later). A 64-bit collision inside the window would drop a distinct message; with 4096 entries that is about 2^-52 per message.

This collapses byte-identical copies only. Messages that carry the same content in different bytes (a different sender field, a timestamp) are not duplicates to this set; that deduplication belongs to the application.

func NewDedupSet added in v1.7.0

func NewDedupSet(entries int, window time.Duration) *DedupSet

NewDedupSet creates a set remembering the last entries distinct payloads (<= 0 selects 4096). A positive window additionally expires an entry once it is older than the window; 0 disables age expiry. Memory is fixed at about 40 bytes per entry.

func (*DedupSet) Entries added in v1.7.0

func (d *DedupSet) Entries() int

Entries returns the number of distinct payloads the set remembers.

func (*DedupSet) Seen added in v1.7.0

func (d *DedupSet) Seen(data []byte) bool

Seen reports whether data is byte-identical to a payload recorded within the set's capacity (and window); if not, it records data and returns false. Safe for concurrent use.

func (*DedupSet) SeenKey added in v1.7.0

func (d *DedupSet) SeenKey(key uint64) bool

SeenKey is Seen for a caller-computed 64-bit key (see SubscriberOptions.DedupKey). Keys from different sources must not be mixed in one set unless they are drawn from the same space.

func (*DedupSet) Window added in v1.7.0

func (d *DedupSet) Window() time.Duration

Window returns the age after which an entry stops counting as seen (0 = never).

type Handler

type Handler func(msg *nats.Msg)

Handler is the subscription message callback.

Concurrency and ordering contract: in worker-pool mode (the default) the callback is invoked concurrently by up to Workers goroutines, and processing order is not guaranteed to match publish order (several workers pull from the same queue), so shared state inside the callback needs its own synchronisation. In go-per-message mode the callback runs on detached goroutines with no upper bound on concurrency, and again without ordering. For strict ordering use Workers: 1 (ordered, delivery never blocks) or Workers: SyncMode (ordered, but blocks the NATS delivery goroutine). Under a mirrored subscription (NewMirroredSubscriber) the callback receives every copy from every endpoint; the ordering guarantees above hold only within a single endpoint, and copies from different endpoints interleave freely. Mirrored x SyncMode warning: the "serial and ordered" property of synchronous mode does not survive mirroring. Each of the M endpoints has its own NATS delivery goroutine, so the callback is invoked concurrently by M goroutines. A callback that relies on SyncMode serialisation to skip locking is a data race under mirroring and must be made concurrency-safe.

In worker-pool and go-per-message modes the callback may do slow work. In synchronous mode it runs on the NATS delivery goroutine, and blocking for too long triggers slow-consumer drops (counted in SlowConsumers).

msg.Data is memory allocated for that message alone (nats.go does not reuse its read buffer), so the callback may retain it after returning (store it, hand it to another goroutine) without a defensive copy.

type Logger

type Logger interface {
	Info(msg string, ctx ...any)
	Warn(msg string, ctx ...any)
	Error(msg string, ctx ...any)
}

Logger is the minimal logging interface the package needs. The standard library's *slog.Logger and go-ethereum's log.Logger both satisfy it as-is: pass slog.Default() or log.Root() directly.

type Options

type Options struct {
	URL   string // Required. Full NATS URL, e.g. nats://token@host:4222 (typically the nearest server or leaf node).
	Topic string // Required. The single topic this connection is bound to.
	Name  string // Client name (shown in server monitoring and logs); defaults to natslink-<role>-<topic>.

	Logger Logger // Log output; nil means slog.Default().

	// ReportInterval is the interval of the built-in status report: every
	// interval one Info line with statistics is logged, plus a Warn line when
	// not connected. Defaults to 1 minute; a negative value disables it, in
	// which case the caller aggregates Stats() itself.
	//
	// The supervision parameters (reconnect backoff, heartbeat, rebuild
	// backoff, ...) are package constants, see the top of natslink.go; they
	// need no configuration.
	ReportInterval time.Duration

	// ExtraNatsOptions are additional options appended to nats.Connect (e.g.
	// TLS configuration, a custom dialer). They are the escape hatch for
	// overriding the underlying policy parameters. Two caveats:
	//   - overriding MaxReconnects / RetryOnFailedConnect / CustomReconnectDelay /
	//     PingInterval / MaxPingsOutstanding removes the corresponding line of
	//     defence from the three-layer failure model; you are on your own;
	//   - the event callbacks (ClosedHandler / ErrorHandler / DisconnectErrHandler /
	//     ...) are injected after these options and cannot be overridden: the
	//     self-healing rebuild and the SlowConsumers observation are the core
	//     promise of this package and there is no way to remove them.
	ExtraNatsOptions []nats.Option
}

Options is the connection configuration shared by Subscriber and Publisher. Zero-valued fields use defaults.

type Publisher

type Publisher struct {
	// contains filtered or unexported fields
}

Publisher is a self-supervising connection dedicated to publishing on a single topic. Usage: NewPublisher -> Start -> Publish(...) -> Stop.

While disconnected, Publish writes into the reconnect buffer (ReconnectBufSize) and returns nil; the buffered messages are flushed automatically once reconnected. It returns an error only when the buffer overflows or the connection is inside a self-healing rebuild, leaving failover or discard to the caller. For round-robin over several connections with failover use PublisherPool.

func NewPublisher

func NewPublisher(o PublisherOptions) (*Publisher, error)

NewPublisher creates a publishing connection (not yet connected; call Start).

func (*Publisher) Conn

func (l *Publisher) Conn() *nats.Conn

Conn returns the current underlying connection; it may be nil inside a rebuild window. It is an escape hatch only (e.g. for Request-Reply); subscribing or publishing through it directly is outside this package's supervision.

func (*Publisher) IsConnected

func (l *Publisher) IsConnected() bool

IsConnected reports whether the connection is currently established (false while RECONNECTING or inside a rebuild window).

func (*Publisher) MustStart

func (p *Publisher) MustStart()

MustStart is Start but panics on failure.

Note: MustStart succeeding does not mean connected (it succeeds even when the server is unreachable; retries continue in the background). For fail-fast "die if it cannot connect" semantics at startup, call MustProbe first (the standard pattern is in the Probe documentation).

func (*Publisher) Publish

func (p *Publisher) Publish(data []byte) error

Publish sends one message on the bound topic. A nil return means the connection accepted the message: sent immediately when connected, or written into the reconnect buffer during RECONNECTING and flushed on reconnect (core NATS has no delivery guarantee, see README). A non-nil error means the message was not accepted. Possible errors (all but the first two pre-check rejections are counted in Stats.Dropped):

ErrStopped    the instance has been stopped (single use; create a new one)
ErrNotRunning Start has not been called
ErrOversized  larger than MaxPayload
ErrNoConn     the connection is inside a rebuild window (old connection permanently closed, new one not yet swapped in)
other         an underlying nats error (e.g. reconnect buffer overflow, nats.ErrReconnectBufExceeded)

Safe for concurrent use. data is copied synchronously into the send/reconnect buffer before return, so the caller may reuse it immediately.

func (*Publisher) Start

func (p *Publisher) Start() error

Start opens the connection. A temporarily unreachable server is not an error (it retries in the background, and Publish writes into the reconnect buffer meanwhile); the returned error is essentially only a configuration error such as a bad URL and should be treated as fatal.

func (*Publisher) Stats

func (l *Publisher) Stats() Stats

Stats returns a snapshot of the current status and data statistics. It is safe to call concurrently at any time.

func (*Publisher) Stop

func (p *Publisher) Stop()

Stop makes a best-effort drain of the pending send buffer (1 second timeout) and then closes the connection. While disconnected the drain necessarily fails and the unsent messages in the reconnect buffer are discarded (a Warn log carries the buffered byte count). Bounded promise: the whole call blocks at most stopGrace (5s); if the peer is wedged the wait is abandoned and the drain and close continue in the background (see link.stop). In that case Stop returning does not mean the connection is closed, and the Warn for a failed flush may come after the return (lost if the process exits right away). Stop is idempotent and safe to call repeatedly; the instance is not reusable afterwards.

func (*Publisher) Topic

func (l *Publisher) Topic() string

Topic returns the topic this connection is bound to.

func (*Publisher) WaitConnected

func (l *Publisher) WaitConnected(timeout time.Duration) error

WaitConnected blocks until the connection is ready, or returns an error on timeout (including the current status and the underlying error, e.g. an authentication failure).

This is a "wait until ready" utility (for tests, or when the application must be online before its first message). For startup fail-fast use MustProbe (see the Probe documentation): one probe connection exposes environment errors without every connection waiting out its own timeout. Also note that on a bad token this method waits the full timeout before returning (auth errors are deliberately treated as recoverable and retried, see the README's three lines of defence, item 2); the last_err in the returned error names the authentication failure.

type PublisherOptions

type PublisherOptions struct {
	Options

	// MaxPayload > 0 rejects messages larger than that many bytes outright
	// (returns ErrOversized and counts towards Dropped).
	// 0 means no application-level limit (the server still enforces its own max_payload).
	MaxPayload int
}

PublisherOptions configures one publishing connection.

type PublisherPool

type PublisherPool struct {
	// contains filtered or unexported fields
}

PublisherPool is a pool of publishers. The plain constructor (NewPublisherPool) builds N connections to one endpoint: round-robin start position, failover one connection at a time, nil as soon as any one of them accepts the message. The mirrored constructors (NewMirroredPublisher, NewMirroredPublisherPool) build connections to each of M independent endpoints: every endpoint is an independent send line with its own bounded queue and goroutine, Publish hands one shared copy of the message to each queue without blocking and returns nil as soon as any queue accepts (multi-entry racing on the send side, deduplication on the consumer side). Both constructors return the same type and are used identically apart from construction. While one connection is in a self-healing rebuild, traffic automatically lands on the remaining connections of that endpoint, so 1/N of the messages are not silently lost.

func NewMirroredPublisher added in v1.7.0

func NewMirroredPublisher(urls []string, o PublisherOptions) (*PublisherPool, error)

NewMirroredPublisher creates one publishing connection per endpoint, at most maxMirroredPublisherEndpoints (5) endpoints, and returns them as a *PublisherPool whose Publish sends once per endpoint and returns nil as soon as any endpoint accepts (see PublisherPool.Publish). Connections are named <Name>-e<endpoint index>. With a single URL there is nothing to mirror: the result is exactly NewPublisherPool(1, o) for that URL (synchronous Publish, connection named <Name>-0), so a configuration can grow from one endpoint to several without changing the host code. With two or more URLs every endpoint is an independent send line (own bounded queue and goroutine), so one stalled endpoint never delays the others. Use NewMirroredPublisherPool when each endpoint needs several connections.

func NewMirroredPublisherPool

func NewMirroredPublisherPool(urls []string, n int, o PublisherOptions) (*PublisherPool, error)

NewMirroredPublisherPool creates, for each of the M endpoints, a group of n publishing connections on the same topic (not yet connected; call Start). Connections are named <Name>-e<endpoint index>-<member index>. Each endpoint is an independent send line (own queue and goroutine); Publish semantics are described in the mirroring paragraph of PublisherPool.Publish. With a single URL there is nothing to mirror and the result is exactly NewPublisherPool(n, o) for that URL (members named <Name>-<i>, synchronous Publish).

func NewPublisherPool

func NewPublisherPool(n int, o PublisherOptions) (*PublisherPool, error)

NewPublisherPool creates n publishing connections on the same topic (not yet connected; call Start). Each connection's client name is <Name>-<i>.

func (*PublisherPool) ConnectedCount

func (p *PublisherPool) ConnectedCount() int

ConnectedCount returns the number of currently connected connections in the pool (for observing degradation).

func (*PublisherPool) Dropped

func (p *PublisherPool) Dropped() uint64

Dropped returns the cumulative count of messages for which every endpoint failed and the whole message was discarded. In a mirrored pool this is settled asynchronously, once the last endpoint's send goroutine has handled the message (or dropped it at the queue). This is the pool's true message-loss count: a single connection's Stats.Dropped is the number of failed attempts on that connection, and the message may well have been failed over to another connection (or, under mirroring, another endpoint) and sent successfully, so summing the members grossly overstates loss (false alarms).

func (*PublisherPool) Endpoints

func (p *PublisherPool) Endpoints() int

Endpoints returns the number of endpoints the pool covers (len(urls) for the mirrored constructor, always 1 for the plain one).

func (*PublisherPool) MustStart

func (p *PublisherPool) MustStart()

MustStart is Start but panics on failure. Suited to fail-fast startup.

func (*PublisherPool) Publish

func (p *PublisherPool) Publish(data []byte) error

Publish sends one message. With the plain constructor it amounts to "two routing passes over the pool, nil as soon as one connection succeeds"; with the mirrored constructor the same routing runs once per endpoint (one send each) and it returns nil as soon as any endpoint accepts. Mirroring is racing redundancy: a single endpoint failing merely loses one path (the message was delivered via the others) and is not counted as loss. Only when every endpoint fails does it return an error (errors.Join) and count towards the pool-level Dropped.

The two routing passes within an endpoint: the first pass tries only members that are currently connected (IsConnected), so the message leaves immediately on a healthy connection and never sits in a disconnected member's reconnect buffer until that member reconnects (latency first: a message delivered seconds late is about as good as undelivered, while a healthy connection was sitting idle right next to it). The second pass turns to the disconnected members: when no member of the endpoint is healthy it falls back to "written into the reconnect buffer counts as success", so the message is buffered rather than dropped.

Mirrored pools (two or more endpoints) do not send on the caller's goroutine at all: one copy of data is handed to every endpoint's bounded send queue (non-blocking) and nil means at least one queue accepted it; the endpoint goroutines then run the routing above independently, so a stalled endpoint only backs up its own queue (its copies are dropped once the queue is full, counted on that endpoint) while the others keep sending. Send failures surface asynchronously in the members' Stats, and a message that every endpoint failed to send still counts towards Dropped. ErrQueueFull is returned when no endpoint could take the message.

Safe for concurrent use. data is copied before Publish returns, so the caller may reuse it immediately.

func (*PublisherPool) Size

func (p *PublisherPool) Size() int

Size returns the total connection count of the pool (mirrored constructor: endpoints x n per endpoint).

func (*PublisherPool) Start

func (p *PublisherPool) Start() error

Start starts every connection in the pool; if any fails (configuration error) the ones already started are stopped and that error is returned.

Members are dialled with bounded concurrency (maxConcurrentDials): a slow handshake on one endpoint's members no longer postpones the dial of an independent endpoint by their count times the dial timeout.

func (*PublisherPool) Stats

func (p *PublisherPool) Stats() []Stats

Stats returns a statistics snapshot per connection in the pool, flattened by member (with the mirrored constructor, endpoint 0's members come first, in urls order; each entry's Name carries the -e<endpoint>-<member> suffix for attribution). Note that each entry's Dropped includes intermediate failover failures; to monitor message loss use the pool-level Dropped() (see its documentation). In a mirrored pool every member of an endpoint also reports that endpoint's send-queue level in QueueLen / QueueCap and the copies its queue discarded in QueueDropped, so a stalled endpoint shows as a filling queue and then as a growing QueueDropped.

func (*PublisherPool) Stop

func (p *PublisherPool) Stop()

Stop stops every connection in the pool. Members are closed in parallel: each blocks at most stopGrace (see link.stop), and the parallelism makes the pool-level worst case max rather than the sum, so the shutdown budget (e.g. systemd TimeoutStopSec) does not shrink linearly with pool size.

func (*PublisherPool) WaitConnected

func (p *PublisherPool) WaitConnected(timeout time.Duration) error

WaitConnected blocks until every connection in the pool is ready; on timeout it returns the error of the first connection that was not ready. Usage is described on Publisher.WaitConnected.

type Stats

type Stats struct {
	Name      string // Connection name (Options.Name; pool members carry a -<i> suffix, mirror endpoints a -e<i> suffix). Used to attribute a degraded connection.
	Running   bool   // true between Start and Stop.
	Connected bool   // Whether currently connected.
	Status    string // Underlying connection status (CONNECTED / RECONNECTING / ...); "no-conn" when there is no connection.
	Server    string // Name of the currently connected server.
	Cluster   string // Name of the currently connected cluster.

	Reconnects uint64 // Successful client automatic reconnects (RECONNECTING -> CONNECTED).
	Rebuilds   uint64 // Whole-connection self-healing rebuilds after a "permanent close" (excluding the initial connect).
	InMsgs     uint64 // Messages received.
	OutMsgs    uint64 // Messages sent.
	InBytes    uint64 // Bytes received.
	OutBytes   uint64 // Bytes sent.

	// Dropped is, for a Publisher, the number of failed publishes (rebuild
	// window / buffer overflow); for a Subscriber, the number of messages
	// dropped by back-pressure because the worker queue was full. Only the
	// worker-pool mode increments it; the synchronous and go-per-message modes
	// keep it at 0 (in synchronous mode the data-loss signal is SlowConsumers;
	// go-per-message never drops and has no back-pressure signal either, see
	// the trade-off notes on SubscriberOptions.Workers).
	// Note: inside a PublisherPool, a single connection's Dropped counts
	// "attempts that failed on this connection"; the message may well have
	// been failed over to another connection and sent. To monitor the pool's
	// real losses use PublisherPool.Dropped(); summing the members' Dropped
	// grossly overestimates loss (false alarms).
	Dropped uint64
	// Paused reports whether the subscriber is paused (after Subscriber.Pause,
	// before Resume): the connection is up but the subscription is gone, so
	// the server delivers nothing to this connection. Always false for a
	// Publisher.
	Paused bool
	// Pauses is the cumulative number of successful Pause transitions on the
	// subscriber (idempotent repeated calls are not counted). Always 0 for a
	// Publisher.
	Pauses uint64
	// Diverged is the number of subscriber endpoints whose subscription state
	// disagrees with the paused state (v1.5.1): not paused but no subscription
	// on the connection (SUB failed during Resume), or paused but the
	// subscription is still there (UNSUB failed during Pause). Only endpoints
	// whose connection is alive are counted (an endpoint inside its rebuild
	// window is handled by rebuild; look at Connected instead). While it is
	// >0 the package's reconcile goroutine is retrying with backoff; if it
	// stays >0 the reconcile is not succeeding, see the "[natslink]
	// subscription reconcile failed" log line. Always 0 for a Publisher.
	Diverged int
	// Gated is the number of messages the subscriber dropped because the gate
	// (SubscriberOptions.Gate returned false) rejected them. The decision is
	// made at the delivery entry point, before the queue and the callback, so
	// all three consumption modes count it; always 0 when no Gate is set.
	// Under a mirrored subscriber it is, like Dropped, a single value shared
	// across endpoints and likewise counted per copy.
	Gated uint64
	// Deduped is the number of messages the subscriber dropped at the delivery
	// entry point because SubscriberOptions.Dedup had already seen a
	// byte-identical payload (one per dropped copy; under mirroring the
	// expected steady-state value is (endpoints-1) per message). Always 0
	// without a DedupSet and for publishers.
	Deduped uint64
	// SlowConsumers is the number of times the subscriber was too slow and the
	// NATS client discarded messages (ErrSlowConsumer). The connection still
	// looks healthy at that point, so this counter is the key signal for
	// detecting "silent data loss" (in synchronous mode it is the only
	// signal). It counts events, not messages, and must not be added to
	// Dropped as a total loss figure.
	SlowConsumers uint64

	QueueLen int // Current length of the Subscriber worker queue (always 0 in synchronous / go-per-message mode); for a mirrored pool member, its endpoint's send-queue length.
	QueueCap int // Capacity of the Subscriber worker queue (always 0 in synchronous / go-per-message mode); for a mirrored pool member, its endpoint's send-queue capacity.
	// QueueDropped is, for a member of a mirrored pool, the number of copies its
	// endpoint's send queue discarded: queue full (entries or bytes) or still
	// queued when Stop gave up on a stalled send goroutine. Send failures are
	// not included (they are this member's Dropped). Always 0 for subscribers
	// and plain publishers.
	QueueDropped uint64
}

Stats is a point-in-time snapshot of status and data statistics. All counters accumulate across self-healing rebuilds. At the instant of a rebuild switch-over a counter may briefly appear to go backwards (the new connection has been swapped in but the old connection's statistics have not been absorbed yet); when computing deltas for monitoring, clamp negative increments to zero.

type Subscriber

type Subscriber struct {
	// contains filtered or unexported fields
}

Subscriber is a self-supervising subscription to a single topic. The plain constructor (NewSubscriber) holds one connection; the mirrored constructor (NewMirroredSubscriber) holds one connection per endpoint sharing a single worker pool and callback, and Handler receives every copy from every endpoint (deduplication is the consumer's job). Both constructors return the same type and are used identically apart from construction. Usage: NewSubscriber / NewMirroredSubscriber -> Start -> (Stats/IsConnected at any time; Pause/Resume to suspend and re-subscribe without disconnecting) -> Stop.

func NewMirroredSubscriber

func NewMirroredSubscriber(urls []string, o SubscriberOptions) (*Subscriber, error)

NewMirroredSubscriber creates one subscribing connection per endpoint (not yet connected; call Start), all sharing a single worker pool and callback: the Handler receives every copy from every endpoint, and deduplication is the caller's job; Workers is the total concurrency bound across endpoints. Connections are named <Name>-e<endpoint index>. Note that SyncMode is no longer serial under a mirror (M delivery goroutines call the callback concurrently, see the Handler comment); the remaining concurrency / ordering / statistics semantics are documented on Handler and Subscriber.Stats. With a single URL there is nothing to mirror and the result is exactly NewSubscriber for that URL (plain name, no -e0).

func NewSubscriber

func NewSubscriber(o SubscriberOptions) (*Subscriber, error)

NewSubscriber creates a subscription connection (not yet connected; call Start).

func (*Subscriber) Conn

func (s *Subscriber) Conn() *nats.Conn

Conn returns the first endpoint's underlying connection, which may be nil during a rebuild window. Escape hatch only (e.g. Request-Reply); under mirroring it exposes only the endpoint for the first url.

func (*Subscriber) Endpoints

func (s *Subscriber) Endpoints() int

Endpoints returns the endpoint count (len(urls) for the mirrored constructor, always 1 for the plain one).

func (*Subscriber) IsConnected

func (s *Subscriber) IsConnected() bool

IsConnected reports whether every endpoint is connected (false while RECONNECTING or inside a rebuild window). Under mirroring any endpoint being down yields false: endpoint degradation must be visible (the race has degraded to a single path).

func (*Subscriber) IsPaused

func (s *Subscriber) IsPaused() bool

IsPaused reports whether the subscription is currently paused (between Pause and Resume).

func (*Subscriber) MustStart

func (s *Subscriber) MustStart()

MustStart is Start but panics on failure.

Note: MustStart succeeding does not mean connected (it succeeds even when the server is unreachable; retries continue in the background). For fail-fast "die if it cannot connect" semantics at startup, call MustProbe first (the standard pattern is in the Probe documentation; for the mirrored constructor probe each endpoint URL separately).

func (*Subscriber) Pause

func (s *Subscriber) Pause() error

Pause suspends the subscription while keeping the connections open: it sends UNSUB on each endpoint's current connection, after which the server stops delivering this topic to the connection. Interest is withdrawn hop by hop along leaf and route links, so upstream NATS servers stop forwarding the stream to this node as well: the saving is real bandwidth, not a local discard (contrast Gate, which drops at the delivery entry point while the bytes still arrive on the NIC). Typical use: a host that knows it will not need this stream for a long stretch. The connection itself, heartbeats, automatic reconnect and self-healing rebuilds all continue, and Stats.Connected is unaffected; meanwhile Stats.Paused is true and Stats.Pauses is incremented once.

Semantics and edges:

  • Idempotent: calling it while already paused returns nil. After Stop it returns ErrStopped.
  • A self-healing rebuild during the pause still swaps in the new connection, it just does not subscribe on it; a nats.go automatic reconnect (RECONNECTING -> CONNECTED) replays only the subscriptions still on the books, so an UNSUBed one does not come back. Neither path needs caller intervention.
  • Messages already in flight before the UNSUB still arrive and are delivered (the server takes a moment to process the UNSUB), and in worker-pool mode messages already queued but not yet consumed are still delivered to the callback; callers that need strict point-in-time semantics must re-check inside Handler (the same rule as Gate's queue residency window).
  • An endpoint inside a rebuild window (connection permanently closed, new one not yet swapped in) has nothing to do and is not an error: the new connection is handled as paused when it is swapped in.
  • The returned error is the join (errors.Join) of per-endpoint Unsubscribe failures; "connection already closed" and "handle already invalid", both meaning there was no subscription to begin with, are not counted as errors.
  • When an endpoint's UNSUB fails, its handle stays on the books and the paused state still takes effect; the package's reconcile goroutine retries with backoff until the UNSUB succeeds (or the connection closes and the rebuild takes over), logging "natslink unsubscribe retried after failed pause"; until it converges Stats.Diverged > 0. The caller does not need to retry.

Unsubscribe takes the underlying connection lock and can block up to flusherTimeout (5s) if the peer is wedged; do not call it synchronously on a latency-sensitive path.

func (*Subscriber) Resume

func (s *Subscriber) Resume() error

Resume re-subscribes: it sends SUB again on each endpoint's most recently set-up connection, after which messages flow to the same Handler / worker pool. Idempotent: returns nil when not paused; returns ErrStopped after Stop.

An endpoint whose connection is permanently closed (inside a rebuild window) is skipped: the setup of the rebuilt connection subscribes it as "not paused", so the caller need not retry. A connection in RECONNECTING has its SUB queued by nats.go and applied on reconnect, which is likewise not an error. The returned error is the join of per-endpoint subscribe failures; when one endpoint fails the others still resume and the paused state is already cleared (Stats.Paused is false). The failed endpoint is retried with backoff by the package's reconcile goroutine until its SUB succeeds (before that existed, an endpoint whose connection stayed healthy and never rebuilt would remain unsubscribed forever), logging "natslink subscribe retried after failed resume"; until it converges Stats.Diverged > 0. The caller does not need to retry.

func (*Subscriber) Start

func (s *Subscriber) Start() error

Start opens every endpoint connection and subscribes. A temporarily unreachable server is not an error (it retries in the background until connected, and the subscription takes effect with the connection). An error from any endpoint (essentially only configuration errors such as a bad URL or topic, which should be treated as fatal) rolls back the endpoints already started.

The worker pool is started before any endpoint is dialled, so an endpoint that connects early never delivers into a queue nobody drains while another endpoint is still dialling; endpoints are dialled with bounded concurrency (maxConcurrentDials), so startup takes the slowest batch, not the sum.

func (*Subscriber) Stats

func (s *Subscriber) Stats() Stats

Stats returns a statistics snapshot merged across endpoints; safe to call concurrently at any time. Merge rules:

  • Name is the first endpoint's connection name (plain constructor: the connection name itself);
  • Connected is true only when every endpoint is connected (endpoint degradation shows up immediately as not connected, so a host's "degraded" alert covers it for free); Status/Server/Cluster come from the first connected endpoint, or from the first endpoint when none is connected;
  • counters (Reconnects/Rebuilds/In*/Out*/SlowConsumers) are summed across endpoints;
  • Dropped is the worker-queue overflow count (the queue is shared, so it is a single value anyway). Note that under mirroring each message has M copies, each taking one queue slot, so under overload Dropped counts copies (roughly M times amplified); content is lost only when every copy of a message is dropped, so divide by the multiplier before panicking about Dropped > 0;
  • for the plain (single-endpoint) constructor every field matches the previous behaviour.

func (*Subscriber) Stop

func (s *Subscriber) Stop()

Stop closes every endpoint connection, ends the worker goroutines, and blocks until all in-flight callbacks have returned. After Stop returns it is safe to release the resources the callback depends on (DB handles, channels, ...) without racing a straggling callback. Bounded promise: closing a single endpoint connection waits at most stopGrace (5s); if the peer is wedged the wait is abandoned and the close continues in the background (see link.stop), so in extreme cases the connection may not be fully closed when Stop returns. The GoPerMessage/SyncMode dispatch carries a done gate, so no new callbacks start after Stop (apart from the single-message race window with a callback already in flight). Callbacks must therefore terminate normally; a permanently blocked callback wedges Stop. Messages still queued and not yet picked up are discarded. The wait guarantee covers worker-pool mode only: in synchronous mode (SyncMode) the callback runs on the NATS delivery goroutine, and in go-per-message mode (GoPerMessage) on detached goroutines, and Stop waits for neither. A go-per-message user that wants to release callback resources after Stop must confirm for itself that in-flight callbacks have finished (or shut down in reverse dependency order so that the dependency outlives this subscription). Stop is idempotent and safe to call repeatedly; the instance is not reusable afterwards.

func (*Subscriber) Topic

func (s *Subscriber) Topic() string

Topic returns the topic this subscription is bound to.

func (*Subscriber) WaitConnected

func (s *Subscriber) WaitConnected(timeout time.Duration) error

WaitConnected blocks until every endpoint connection is ready; on timeout it returns the error of the first endpoint that was not ready. Usage and caveats are in the link.WaitConnected / Probe documentation.

type SubscriberOptions

type SubscriberOptions struct {
	Options

	Handler Handler // Required. Message callback (concurrency contract on the Handler type).

	// Gate is a callback admission check. When non-nil it is called once per
	// message at the delivery entry point; the message continues to Handler only
	// if it returns true, otherwise it is dropped on the spot and counted
	// (Stats.Gated). nil means always admit (the previous behaviour). Typical
	// use: a host that is not yet ready to process messages (still warming up,
	// still catching up on state) gates everything off, and admission resumes
	// automatically once it is ready, without rebuilding the subscription.
	//
	// Contract: Gate runs on the NATS delivery goroutine (under mirroring, on
	// each endpoint's own delivery goroutine, concurrently), so it must be
	// concurrency-safe and extremely cheap (an atomic read; a slow Gate blocks
	// delivery and triggers slow-consumer drops). The decision is made before
	// enqueue/dispatch: in worker-pool mode a message that was admitted and
	// queued, but not yet picked up by a worker when the gate closes, is still
	// delivered to the callback (queue residency window). Callers that need
	// strict point-in-time semantics must re-check inside Handler.
	Gate func() bool

	// Dedup, when non-nil, drops a message whose payload is byte-identical to
	// one the set has recorded recently (Stats.Deduped) before it is queued or
	// dispatched. It is a general byte-level filter for any subscriber: the
	// same set may be given to one subscriber, to several (identical payloads
	// on different subjects then collapse as well, e.g. one event published on
	// three feeds), or to every subscriber a process owns, and the host may
	// also call DedupSet.Seen directly on paths that do not go through a
	// Subscriber. The typical case is a mirrored subscription, where the M
	// endpoint copies of one message collapse into a single callback without
	// decoding and the first copy to arrive wins. nil means every message is
	// delivered (the previous behaviour).
	//
	// The check runs after Gate and before the dispatch mode, on the NATS
	// delivery goroutine: one hash of the payload (outside the set's lock) plus
	// a short locked map lookup. Messages with an empty payload bypass it.
	// Only byte-identical copies are collapsed; two messages carrying the same
	// content in different bytes are not duplicates to the set, so an
	// application that needs content-level deduplication keeps it. The window
	// is a capacity (and optionally an age), not a guarantee: a copy arriving
	// after the set has forgotten the first one is delivered again.
	Dedup *DedupSet

	// DedupKey, when non-nil, replaces the payload hash with a caller-computed
	// 64-bit key for the Dedup check (for example a hash already present in the
	// payload at a fixed offset, saving the hash of a large message). Returning
	// ok=false delivers the message without consulting the set. Requires Dedup;
	// runs on the delivery goroutine, so it must be cheap and concurrency-safe.
	DedupKey func(msg *nats.Msg) (key uint64, ok bool)

	// Workers is the worker-pool goroutine count (bounded concurrency), default 8.
	// The callback never blocks the NATS delivery goroutine; when the pool cannot
	// keep up, new messages are dropped and counted (Stats.Dropped, so backpressure
	// is visible), which avoids the goroutine explosion / OOM that go-per-message
	// suffers under backpressure.
	// Workers = SyncMode (-1) selects synchronous mode: no worker pool, the
	// callback runs directly on the NATS delivery goroutine. Lowest latency (one
	// queue hand-off saved) but only suitable for non-blocking, very light
	// processing (a callback with lock contention, IO or millisecond-scale compute
	// belongs in the worker pool).
	// Workers = GoPerMessage (-2) selects go-per-message mode: no worker pool, the
	// delivery goroutine spawns a new goroutine per message, with unbounded
	// concurrency. Suited to streams where most messages are light but a few do
	// heavy synchronous work and must not be dropped. A bounded pool on such mixed
	// traffic gets its workers filled by the heavy messages, head-of-line blocks
	// the light ones, and drops under bursts, while the throughput ceiling of such
	// callbacks is usually a downstream lock, so extra goroutines merely queue on
	// it. The cost (read before choosing it): this mode has no backpressure signal
	// at all (Dropped stays 0; the callback never blocks delivery, so
	// SlowConsumers hardly ever fires either), the goroutine count floats with
	// load, and the caller must know their upstream rate and downstream capacity.
	// Any other negative value is a configuration error (NewSubscriber returns an error).
	// A mirrored subscription shares one worker pool: Workers is the total
	// concurrency bound across endpoints, not multiplied per endpoint.
	Workers int
	// QueueSize is the worker-pool buffer length (backpressure bound), default 4096.
	// No negative-value semantics; <= 0 always means the default.
	// A mirrored subscription shares one queue.
	QueueSize int

	// QueueGroup, when non-empty, makes this a queue subscription: subscribers in
	// the same group load-balance and compete for messages (each message goes to
	// one member). Unset means broadcast: every Subscriber receives every message.
	// For replicated deployments leave it unset to process everything on every
	// replica, set it to share the load.
	// Under mirroring the queue-group semantics apply per endpoint: members of a
	// group compete only within one endpoint, and each endpoint still delivers
	// its own copy.
	QueueGroup string

	// NATS client subscription buffer limits (another layer of protection before
	// this package's callback). When exceeded the client drops messages and raises
	// ErrSlowConsumer (counted in Stats.SlowConsumers). Essentially only reached in
	// synchronous mode with a slow callback (or an extreme burst): in worker-pool
	// mode backpressure is governed by Workers/QueueSize, and these two normally
	// need no tuning. Under mirroring they apply per endpoint.
	PendingMsgLimit   int // 0 uses the nats default (512k messages); -1 means unlimited
	PendingBytesLimit int // 0 uses the nats default (64MB); -1 means unlimited
}

SubscriberOptions configures one subscription connection.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL