The highest tagged major version is
Documentation
¶
Overview ¶
Package simnet provides a seeded, fault-injecting simulation of the two seams a Raft node talks to: the network (raft.Transport) and stable storage (raft.Storage).
It exists because the plain in-memory transport delivers every RPC by a direct, immediate function call. That makes tests fast but it also means the only failure a test can express is "this link is down". Real networks lose, duplicate, delay and reorder individual messages, and real disks fail; a test suite that cannot express those faults cannot claim to have exercised the parts of Raft that exist to survive them.
Determinism ¶
Every decision the simulator makes — whether a message is lost, whether it is duplicated, how long each leg of the round-trip takes — is drawn from a single math/rand/v2 generator seeded from one uint64. Re-running with the same seed replays the same decision sequence for the same sequence of submitted messages.
That is reproducibility in practice, not in principle, and the distinction matters. The Raft node under test is a multi-goroutine program driven by a real clock: its event loop, its per-peer heartbeat pumps, its apply loop and its snapshot workers are scheduled by the Go runtime, not by this package. The simulator therefore cannot control the *order in which messages are submitted to it*, only what happens to each message once submitted. Two runs with the same seed will usually, but not always, follow the same path. Achieving full determinism would require the Raft engine itself to be restructured around a single-threaded, injectable scheduler.
In exchange for that caveat, a seed is cheap and usually enough: a failure found by a randomized test is far more likely to reproduce from its seed than from nothing at all. Tests in this repository print the seed on failure and accept one from the RAFT_SIM_SEED environment variable.
Simulated time ¶
Latency is simulated by sleeping the calling goroutine, so it is wall-clock time and should be kept to single-digit milliseconds. The Raft node's logical clock is separate: tests drive it with Node.Tick (Config.TickInterval == 0).
Index ¶
- Variables
- type FaultStore
- func (f *FaultStore) AllowWrites()
- func (f *FaultStore) AppendLogEntries(ctx context.Context, entries []raft.LogEntry) error
- func (f *FaultStore) Close() error
- func (f *FaultStore) Crash(ctx context.Context) error
- func (f *FaultStore) FailWrites()
- func (f *FaultStore) FirstIndex() (raft.Index, error)
- func (f *FaultStore) GetLogEntries(ctx context.Context, lo, hi raft.Index) ([]raft.LogEntry, error)
- func (f *FaultStore) GetLogEntry(ctx context.Context, index raft.Index) (raft.LogEntry, error)
- func (f *FaultStore) Inner() raft.Storage
- func (f *FaultStore) LastIndex() (raft.Index, error)
- func (f *FaultStore) LoadHardState(ctx context.Context) (raft.HardState, error)
- func (f *FaultStore) LoadSnapshot(ctx context.Context) (raft.SnapshotMeta, io.ReadCloser, error)
- func (f *FaultStore) SaveHardState(ctx context.Context, hs raft.HardState) error
- func (f *FaultStore) SaveSnapshot(ctx context.Context, meta raft.SnapshotMeta, r io.Reader) error
- func (f *FaultStore) SetBuffered(b bool)
- func (f *FaultStore) SnapshotIndex() raft.Index
- func (f *FaultStore) Stats() (writes, failures int)
- func (f *FaultStore) Sync()
- func (f *FaultStore) TruncatePrefix(ctx context.Context, toIndex raft.Index) error
- func (f *FaultStore) TruncateSuffix(ctx context.Context, fromIndex raft.Index) error
- type LinkPolicy
- type Message
- type Network
- func (n *Network) Close() error
- func (n *Network) Cut(from, to raft.NodeID)
- func (n *Network) FaultSchedule() string
- func (n *Network) Heal(id raft.NodeID)
- func (n *Network) HealAll()
- func (n *Network) Isolate(id raft.NodeID)
- func (n *Network) NewTransport(id raft.NodeID) *Transport
- func (n *Network) RecentEvents(limit int) string
- func (n *Network) RecordFault(format string, args ...any)
- func (n *Network) Register(id raft.NodeID, handler raft.Handler)
- func (n *Network) Restore(from, to raft.NodeID)
- func (n *Network) Seed() uint64
- func (n *Network) SetDefaultPolicy(p LinkPolicy)
- func (n *Network) SetFilter(f func(Message) bool)
- func (n *Network) SetLinkPolicy(from, to raft.NodeID, p LinkPolicy)
- func (n *Network) Unregister(id raft.NodeID)
- type RPC
- type SkewClock
- type Transport
- func (t *Transport) AppendEntries(ctx context.Context, to raft.NodeID, req *raft.AppendEntriesRequest) (*raft.AppendEntriesResponse, error)
- func (t *Transport) Close() error
- func (t *Transport) InstallSnapshot(ctx context.Context, to raft.NodeID, req *raft.InstallSnapshotRequest) (*raft.InstallSnapshotResponse, error)
- func (t *Transport) ReadIndex(ctx context.Context, to raft.NodeID, req *raft.ReadIndexRequest) (*raft.ReadIndexResponse, error)
- func (t *Transport) Register(id raft.NodeID, handler raft.Handler)
- func (t *Transport) RequestVote(ctx context.Context, to raft.NodeID, req *raft.RequestVoteRequest) (*raft.RequestVoteResponse, error)
- func (t *Transport) TimeoutNow(ctx context.Context, to raft.NodeID, req *raft.TimeoutNowRequest) (*raft.TimeoutNowResponse, error)
- func (t *Transport) Unregister(id raft.NodeID)
Constants ¶
This section is empty.
Variables ¶
var ( // ErrDropped is returned when the request or its response was lost. ErrDropped = errors.New("simnet: message lost") // ErrUnreachable is returned when no handler is registered for the // destination, which models a node that is down. ErrUnreachable = errors.New("simnet: destination unreachable") // ErrClosed is returned once the Network has been closed. ErrClosed = errors.New("simnet: network closed") )
Errors reported to the Raft node when the simulator decides an RPC does not complete. They are deliberately opaque: a Raft node must treat any RPC failure the same way, so the tests do not let it distinguish them.
var ErrDiskFailure = errors.New("simnet: simulated storage failure")
ErrDiskFailure is the error a FaultStore returns from every mutating call while write failures are armed.
Functions ¶
This section is empty.
Types ¶
type FaultStore ¶
type FaultStore struct {
// contains filtered or unexported fields
}
FaultStore wraps a raft.Storage so a test can take the disk away from a node.
It supports three faults, in increasing order of nastiness:
FailWrites makes every mutating call return an error. This is a legal fault: real disks fill up and fail, and a Raft node is expected to cope (by refusing to acknowledge anything it could not persist) rather than to lie about what it has on stable storage.
Crash rolls the store back to the last durable point and is the basis of crash-restart testing. With the default write-through behaviour every returned write is durable, so Crash is a clean power-cut: the node is stopped, the store keeps exactly what it acknowledged, and a fresh Node is built on top of it.
SetBuffered(true) makes the store acknowledge writes it has not made durable, so a subsequent Crash loses the tail. This models storage that lies about fsync. Raft's safety proof assumes it never does, so enabling this is a way to demonstrate what a lying disk costs, not a way to test Raft: a safety violation found with buffering on is a property of the disk, not of the consensus implementation.
Snapshot writes and prefix truncation are always treated as durable. Both are checkpoints of state that is already durable in the log, so rolling them back would only model a fault that cannot lose committed data.
Safe for concurrent use.
func NewFaultStore ¶
func NewFaultStore(inner raft.Storage) *FaultStore
NewFaultStore wraps inner. The wrapper assumes inner is in a fully durable state at construction time, which is true both for a fresh store and for one being reopened after a crash.
func (*FaultStore) AllowWrites ¶
func (f *FaultStore) AllowWrites()
AllowWrites disarms write failures.
func (*FaultStore) AppendLogEntries ¶
func (*FaultStore) Close ¶
func (f *FaultStore) Close() error
Close does not close the wrapped store: a crash-restart cycle reuses it, and the test that created it owns its lifetime.
func (*FaultStore) Crash ¶
func (f *FaultStore) Crash(ctx context.Context) error
Crash discards everything that was written but not made durable, leaving the store exactly as a power cut would. The caller must have stopped the Node first; restarting means building a new Node on this same store.
With write-through behaviour (the default) nothing is lost and Crash only asserts that fact.
func (*FaultStore) FailWrites ¶
func (f *FaultStore) FailWrites()
FailWrites arms write failures. Every subsequent SaveHardState, AppendLogEntries, TruncateSuffix, TruncatePrefix and SaveSnapshot returns ErrDiskFailure until AllowWrites is called. Reads keep working, which is what a full disk or a read-only remount looks like.
func (*FaultStore) FirstIndex ¶
func (f *FaultStore) FirstIndex() (raft.Index, error)
func (*FaultStore) GetLogEntries ¶
func (*FaultStore) GetLogEntry ¶
func (*FaultStore) Inner ¶
func (f *FaultStore) Inner() raft.Storage
Inner returns the wrapped store. Tests read log contents through it, which deliberately bypasses the injected faults: the invariant checker wants to know what is actually on disk, not what the node is currently allowed to see.
func (*FaultStore) LoadHardState ¶
func (*FaultStore) LoadSnapshot ¶
func (f *FaultStore) LoadSnapshot(ctx context.Context) (raft.SnapshotMeta, io.ReadCloser, error)
func (*FaultStore) SaveHardState ¶
func (*FaultStore) SaveSnapshot ¶
func (f *FaultStore) SaveSnapshot(ctx context.Context, meta raft.SnapshotMeta, r io.Reader) error
func (*FaultStore) SetBuffered ¶
func (f *FaultStore) SetBuffered(b bool)
SetBuffered controls whether acknowledged writes are durable. See the type documentation for why turning this on changes what a test result means.
func (*FaultStore) SnapshotIndex ¶
func (f *FaultStore) SnapshotIndex() raft.Index
SnapshotIndex returns the last index covered by the store's snapshot, or 0 when there is none. It exists so the invariant checker can tell "this node never had that entry" from "this node compacted that entry away".
func (*FaultStore) Stats ¶
func (f *FaultStore) Stats() (writes, failures int)
Stats returns the number of mutating calls made and the number of those that were failed by injection.
func (*FaultStore) TruncatePrefix ¶
func (*FaultStore) TruncateSuffix ¶
type LinkPolicy ¶
type LinkPolicy struct {
// Loss is the probability that a request is lost. The response leg is lost
// with the same probability, so the observable end-to-end failure rate is
// roughly 2×Loss.
Loss float64
// Duplicate is the probability that a delivered request is delivered a
// second time. The duplicate's response is discarded, exactly as a real
// retransmission's would be by the sender's already-satisfied call.
Duplicate float64
// MinLatency and MaxLatency bound the per-leg delay.
MinLatency, MaxLatency time.Duration
// Reorder is the probability that a request is additionally held for
// ReorderDelay, overtaking later messages on the same link.
Reorder float64
ReorderDelay time.Duration
}
LinkPolicy describes the behaviour of one directed link. The zero value is a perfect link: no loss, no duplication, no delay.
Loss and Duplicate are probabilities in [0, 1]. Latency for each leg of the round-trip is drawn uniformly from [MinLatency, MaxLatency); with probability Reorder an extra ReorderDelay is added to the request leg, which is what actually causes two messages sent in one order to arrive in the other.
func Flaky ¶
func Flaky() LinkPolicy
Flaky returns a policy that loses, duplicates, delays and reorders messages at rates high enough to shake out retry and idempotency bugs while still letting a cluster make progress.
func Lossless ¶
func Lossless() LinkPolicy
Lossless returns a policy with a small uniform latency and no faults. It is the sensible default for a test that wants realistic message interleaving without any actual failures.
type Message ¶
type Message struct {
// Seq is a network-wide monotonically increasing submission number.
Seq uint64
// From and To are the sending and receiving node identities.
From, To raft.NodeID
// RPC says which of the five Raft RPCs this is.
RPC RPC
// Req is the concrete request: *raft.RequestVoteRequest,
// *raft.AppendEntriesRequest, *raft.InstallSnapshotRequest,
// *raft.TimeoutNowRequest or *raft.ReadIndexRequest.
Req any
}
Message is a single in-flight RPC as seen by the simulator. It is passed to the filter installed with Network.SetFilter so that a test can make delivery depend on the content of the message — for example, dropping only the AppendEntries RPCs that carry an entry from the leader's own term, which is how the Figure 8 scenario is constructed.
type Network ¶
type Network struct {
// contains filtered or unexported fields
}
Network is the central simulator. Every RPC sent by every Transport it creates passes through it, and every decision about that RPC's fate is drawn from the Network's single seeded generator.
Safe for concurrent use.
func (*Network) Close ¶
Close shuts the Network down and waits for any duplicate-delivery goroutines to finish, so that no simulated message can touch a node after a test has returned.
func (*Network) Cut ¶
Cut takes down the directed link from → to. The reverse direction is left alone, so Cut is how an asymmetric partition is built: after Cut(a, b), a's RequestVote RPCs never reach b but b's heartbeats still reach a.
func (*Network) FaultSchedule ¶
FaultSchedule returns the ordered list of faults injected so far, one per line. This is the second half of a reproduction: the seed says how the simulator will behave, the schedule says what the test did to it.
func (*Network) NewTransport ¶
NewTransport returns a raft.Transport bound to this Network under the identity id.
func (*Network) RecentEvents ¶
RecentEvents returns up to limit of the most recent per-message events the simulator considered noteworthy (losses, duplicates, reorder spikes).
func (*Network) RecordFault ¶
RecordFault appends a line to the fault schedule. Tests call it when they crash a node or otherwise inject a fault the Network itself cannot see, so that the schedule printed on failure is complete.
func (*Network) Seed ¶
Seed returns the value the Network was constructed with. Print it on failure; it is the only thing a maintainer needs to replay the run.
func (*Network) SetDefaultPolicy ¶
func (n *Network) SetDefaultPolicy(p LinkPolicy)
SetDefaultPolicy sets the policy used by every link that has no specific policy of its own.
func (*Network) SetFilter ¶
SetFilter installs a content-aware delivery predicate. Returning false drops the message. A nil filter (the default) delivers everything the policy and the cut set allow. The filter runs while the Network lock is held, so it must not call back into the Network.
func (*Network) SetLinkPolicy ¶
func (n *Network) SetLinkPolicy(from, to raft.NodeID, p LinkPolicy)
SetLinkPolicy overrides the policy for the directed link from → to.
func (*Network) Unregister ¶
Unregister makes id unreachable, which is how a stopped node looks to its peers.
type RPC ¶
type RPC uint8
RPC identifies which Raft RPC a simulated message carries.
The five RPCs of the raft.Transport interface.
type SkewClock ¶
type SkewClock struct {
// contains filtered or unexported fields
}
SkewClock is a raft.Clock whose reading differs from real time by a constant offset and whose rate differs by a constant factor.
Leader leases are the one part of Raft that trades a network round-trip for an assumption about clocks, so they are the one part whose safety argument depends on how far apart two nodes' clocks can drift. Giving each node its own SkewClock is how a test states that assumption out loud and then breaks it on purpose.
Rate < 1 makes the clock run slow, which is the dangerous direction for a leader: a slow clock makes a lease that has really expired still look valid. Rate > 1 makes it run fast, which only costs availability.
Safe for concurrent use.
func NewSkewClock ¶
NewSkewClock returns a clock that starts at the current instant, advances at rate times real speed, and reads offset ahead of (or, when negative, behind) where it would otherwise be.
type Transport ¶
type Transport struct {
// contains filtered or unexported fields
}
Transport is one node's view of the simulated network. It satisfies raft.Transport.
func (*Transport) AppendEntries ¶
func (t *Transport) AppendEntries(ctx context.Context, to raft.NodeID, req *raft.AppendEntriesRequest) (*raft.AppendEntriesResponse, error)
func (*Transport) Close ¶
Close is a no-op: the Network owns the lifetime, not the per-node transport.
func (*Transport) InstallSnapshot ¶
func (t *Transport) InstallSnapshot(ctx context.Context, to raft.NodeID, req *raft.InstallSnapshotRequest) (*raft.InstallSnapshotResponse, error)
func (*Transport) ReadIndex ¶
func (t *Transport) ReadIndex(ctx context.Context, to raft.NodeID, req *raft.ReadIndexRequest) (*raft.ReadIndexResponse, error)
func (*Transport) RequestVote ¶
func (t *Transport) RequestVote(ctx context.Context, to raft.NodeID, req *raft.RequestVoteRequest) (*raft.RequestVoteResponse, error)
func (*Transport) TimeoutNow ¶
func (t *Transport) TimeoutNow(ctx context.Context, to raft.NodeID, req *raft.TimeoutNowRequest) (*raft.TimeoutNowResponse, error)
func (*Transport) Unregister ¶
Unregister makes id unreachable on the underlying Network.
Source Files