wardenpoint

wardenpoint-go

Send alerts and heartbeats to WardenPoint from long-running Go services and devices: a Raspberry Pi in a boiler room, an edge gateway on LTE, a daemon in a container. Standard library only.

Two promises shape the design:

• The client never gets in your service's way. Fire and Resolve only put an event on a bounded in-memory queue; they never wait for the network. The queue's memory use is bounded. A dead WardenPoint, a dead network or a hung connection cost you nothing but log lines.
• The client never loses an alert silently. Every event ends up delivered, rejected by the server (logged at ERROR with the HTTP status and the start of the response), or evicted by queue overflow (logged at WARN by Run's goroutine, so Fire itself never writes a log line). Each outcome is counted in metrics and in Status().

Install

go get github.com/WardenPoint/wardenpoint-go

Go 1.26 or newer (CI covers the two latest releases). Builds for linux/arm, linux/arm64 and anything else Go supports.

Quick start

Create a Generic webhook integration in the dashboard (Observability → Integrations) and copy its inbound URL and secret.

wp, err := wardenpoint.New(wardenpoint.Config{
	IntegrationURL:    "https://wardenpoint.com/api/v1/integrations/alerts/<uuid>",
	Secret:            os.Getenv("WARDENPOINT_SECRET"),
	HeartbeatInterval: time.Minute,
	Logger:            slog.Default(),
})
if err != nil {
	log.Fatal(err)
}
go wp.Run(ctx) // delivery and heartbeat; returns when ctx is done

wp.Fire(wardenpoint.Alert{
	ID:          "boiler-silent",
	Title:       "Boiler sensor silent",
	Description: "No reading for 3m.",
	Severity:    wardenpoint.Critical,
	Labels:      map[string]string{"service": "boiler"},
})
// ... later, when the condition clears:
wp.Resolve("boiler-silent")

A complete service

examples/boiler is a runnable service that uses every piece together. It covers signal handling and a /healthz endpoint that exposes Status(). It also shows a HeartbeatCheck that withholds the heartbeat when the sensor bus is dead, and an alert raised once and resolved once per incident. The core of it:

func main() {
	logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	wp, err := wardenpoint.New(wardenpoint.Config{
		IntegrationURL:    os.Getenv("WARDENPOINT_URL"),
		Secret:            os.Getenv("WARDENPOINT_SECRET"),
		HeartbeatURL:      os.Getenv("WARDENPOINT_HEARTBEAT_URL"),
		HeartbeatSecret:   os.Getenv("WARDENPOINT_HEARTBEAT_SECRET"),
		HeartbeatInterval: time.Minute,
		HeartbeatCheck:    sensor.ping, // no heartbeat while the sensor is dead
		QueueSize:         50,
		Logger:            logger,
		AllowDisabled:     true, // no-op client when the env is empty (local runs)
	})
	if err != nil {
		logger.Error("wardenpoint config", "error", err)
		os.Exit(1)
	}

	done := make(chan struct{})
	go func() { defer close(done); _ = wp.Run(ctx) }()

	http.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
		_ = json.NewEncoder(w).Encode(wp.Status())
	})
	go http.ListenAndServe("127.0.0.1:8080", nil)

	watch(ctx, wp) // Fire on the transition to "silent", Resolve on the way back
	<-done
}

API

Resolve(id) needs the title because WardenPoint stores the title of every event on the alert. A resolve without one would rename the alert to "External alert". The client remembers title, severity, labels and source URL of the last 1024 fired IDs (descriptions are not kept). After a restart, Resolve(id) uses the ID as the title; call ResolveAlert if that matters.

Configuration

Delivery

The body is the generic webhook format: external_alert_id, title, description, severity, status, labels, source_url. It also carries client_event_id, a random ID per event that WardenPoint ignores; it keeps every request body unique. Title and description are truncated to the server's limits (255 and 5000 characters) on Fire, so the server never rejects an alert for its length.

Events are delivered strictly in order, one at a time. While the head of the queue waits to retry, nothing overtakes it.

When the queue is full, the oldest event that is not on the wire at that moment is evicted, with a WARN naming its alert_id.

An event delivered more than LateThreshold after Fire gets a line like Raised at 2026-10-18T12:00:00Z, delivered 7m12s late. appended to its description. Whoever reads it at 3 a.m. then knows the incident is older than the notification.

A 202 whose queue_skip_reason means nobody will be told (no_group, group_has_no_recipients, no_routable_contacts, …) is logged at WARN rather than DEBUG. The alert arrived, but the integration's routing sends it nowhere.

Repeated alerts are not deduplicated

Every Fire is a new event of the alert ID. WardenPoint groups events by ID, so you get one alert, not many. Each event still counts towards the integration's accumulation threshold ("page after N events in M minutes"). The client deliberately does not deduplicate. Fire when the condition starts, not on every poll while it lasts, unless repeated events are what your threshold expects.

A request whose response was lost after the server accepted it (a timeout) is retried, and the event is counted twice. Avoiding that needs an idempotency key on the server side; until then, a duplicate is the cost of never losing an alert.

Heartbeat

The heartbeat runs in its own goroutine. It does not share the alert queue, so an alert stuck in retries or rejected by the server never stops it. The first beat goes out as soon as Run starts. WardenPoint ignores an integration that has never sent anything, so until then a dead device goes unnoticed.

Each beat is a fixed alert:

{
  "external_alert_id": "wardenpoint-heartbeat",
  "title": "Heartbeat",
  "severity": "info",
  "status": "firing",
  "description": "Liveness signal from wardenpoint-go. Suppress it with a routing rule on label kind=heartbeat.",
  "labels": { "kind": "heartbeat" },
  "client_event_id": "<random>"
}

The client logs only when the heartbeat changes state: it starts or stops reaching WardenPoint, or HeartbeatCheck starts failing. An hour-long outage produces one WARN, not sixty.

Setting it up in WardenPoint

1. Turn on heartbeat mode for the integration that receives the beats: health check mode Heartbeat, a grace window (default 180 s, at least 3× your interval), and an owner group. Without an owner group nobody is paged.
2. Suppress the heartbeat alert itself with a routing rule. Otherwise every beat is an incoming alert that may page someone.

Use a dedicated integration per device for heartbeats (HeartbeatURL, HeartbeatSecret). The health check only looks at when the integration last received anything. On a shared integration every real alert counts as a sign of life, and on a shared heartbeat integration one live device hides a dead one. A dedicated integration also needs nothing more than a catch-all suppress rule. If you do send both to one integration, add the rule below above your other rules.

The rule through the configuration API (POST /api/v1/observability/integrations/{integration}/routing-rules, Authorization: Bearer <API key with config.write>):

{
  "name": "Swallow wardenpoint-go heartbeats",
  "label_matchers": [{ "key": "kind", "op": "=", "value": "heartbeat" }],
  "recipient_group_uuid": null,
  "threshold_count": 1,
  "window_minutes": 1,
  "below_threshold_action": "suppress",
  "threshold_action": "suppress",
  "after_threshold_action": "suppress"
}

State all five accumulation fields. Once any of them is set, the rule stops using the integration's accumulation rules: an omitted threshold and window fall back to 1, an omitted after_threshold_action to suppress, and only the two other actions inherit the integration's default action. A suppressed heartbeat still refreshes the integration's "last received" time, and that time is all the health check looks at.

How a missed heartbeat reaches people

WardenPoint checks heartbeats every two minutes, so a page comes between the grace window and about two minutes after it. When no beat arrived within the grace window, it sends one notification to the integration's owner group at priority high: Integration "…" has missed its heartbeat for over N min. When beats resume, it sends heartbeat restored, also at priority high. Routing rules play no part in this: both notifications go straight to the owner group.

HeartbeatCheck: alive is not the same as healthy

HeartbeatCheck: func(ctx context.Context) error {
	return db.PingContext(ctx)
},

When the check returns an error (or panics, or exceeds RequestTimeout), the beat is skipped and Status().Heartbeat.State becomes unhealthy. WardenPoint then pages the owner group once the grace window runs out. This is a choice: the process is up, but you are saying it is not doing its job. Leave the check nil to report liveness only.

Severity is not priority

Severity describes the alert. It does not decide how loudly people are woken. The action of the routing rule that matches the alert decides that:

Severity is one of the conditions a rule can match on (severities: ["critical"]). So "critical alerts call the on-call engineer" is something you configure in a rule; sending Critical alone does not make it happen.

Security

• The secret is sent only as Authorization: Bearer …, or not at all with HMAC: true. It never goes in the URL.
• It never appears in logs, errors, Status() or metrics. That holds for network errors too: a *url.Error from net/http prints the full URL (including any ?token= you put there), so the client replaces it with a redacted message and does not wrap the original. Response bodies quoted in logs are masked as well. Config prints with secrets masked through %v, %+v, %#v and slog.
• A panic in HeartbeatCheck is recovered. Its message is masked and reported as a failed check.
• Plain http:// to a non-loopback host logs a WARN at New.

HMAC

With the integration's Require HMAC setting on, set HMAC: true. Each request then carries X-WardenPoint-Timestamp (Unix seconds) and X-WardenPoint-Signature: hex HMAC-SHA256 of timestamp + "." + body, keyed by the secret. The secret itself is not sent. WardenPoint refuses a signature it has already seen. The random client_event_id keeps two identical alerts fired in the same second from colliding, and a retry is signed afresh.

Timestamps must be within the integration's tolerance of the server's clock. Devices without a real-time clock need NTP before the first request. When a signed request gets 401 and the server's Date is more than a minute off, the error says the device clock is off.

Surviving restarts

By default the queue lives in memory. Run logs a WARN with the count of events that will be lost if the process exits. To keep them, implement Store:

type Store interface {
	Load(ctx context.Context) ([]Event, error)   // once, oldest first
	Add(ctx context.Context, ev Event) error
	Delete(ctx context.Context, eventID string) error
}

Event marshals to JSON as is. The client calls the store only from its delivery goroutine, never concurrently and never from Fire. A slow SD card therefore cannot block your service. As a result, an event fired a moment before a crash may not have reached the store yet. Store failures are logged and do not stop delivery. A failed Delete can only cause a duplicate after a restart, never a loss.

Metrics

type Metrics interface {
	EventDelivered()
	EventDropped(reason DropReason) // "overflow" or "rejected"
	EventRetried()
	QueueLength(n int)
	Heartbeat(ok bool)
}

A Prometheus adapter is three counters, a counter vector by reason, and two gauges. No dependency on any metrics library is imposed. Calls must be cheap and non-blocking.

Logs

slog with the fields alert_id, severity, status, http_status, attempt, queue_len, retry_in, plus error where relevant.

Testing your code

wardenpointtest has a fake server and a manual clock:

srv := wardenpointtest.NewServer(t)
clock := wardenpointtest.NewClock(time.Now())
srv.Script("alerts", wardenpointtest.Reply{Status: 429, Header: http.Header{"Retry-After": {"30"}}})
srv.SetDefault("heartbeat", wardenpointtest.Reply{Hang: true})

wp, _ := wardenpoint.New(wardenpoint.Config{
	IntegrationURL: srv.IntegrationURL("alerts"),
	HeartbeatURL:   srv.IntegrationURL("heartbeat"),
	Secret:         "test",
	Clock:          clock,
})
go wp.Run(ctx)
wp.Fire(wardenpoint.Alert{ID: "x"})

srv.WaitRequests(ctx, "alerts", 1)
clock.WaitTimers(ctx, 1)    // the retry is parked on the clock
clock.Advance(30 * time.Second)
reqs, _ := srv.WaitRequests(ctx, "alerts", 2)

Reply supports a status, headers and a body. It can also hang until the client gives up (Hang) or until you release it (Release). Requests are recorded with headers, raw body and decoded payload. RequireAuth makes an integration check the Bearer secret or the HMAC signature (timestamp window, replayed signatures refused) the way WardenPoint does.

The library's own tests run under go test -race, check for goroutine leaks with goleak (a test-only dependency), and cover:

• a hung server;
• a stuck or rejected queue alongside a live heartbeat;
• ordering, overflow and Retry-After;
• HMAC signatures and replay;
• secret masking on network errors and tokenised URLs.

Known limitations of the WardenPoint API

These are worked around on the client side; server-side support is planned:

• No dedicated heartbeat endpoint: the heartbeat is an alert that a rule has to suppress, and it spends the inbound rate limit (60 requests per minute per source IP, shared by alerts and heartbeats).
• "Heartbeat missed" and "heartbeat restored" both arrive at priority high.
• No Idempotency-Key: a retry after a lost response counts the event twice.
• The 202 response has no notification ID to check delivery against.

License

MIT