Documentation
¶
Overview ¶
Package backend tracks a set of inference backends grouped by the models they serve, health-checks them, and hands out the next healthy one for a given model in round-robin order.
Index ¶
Constants ¶
const ( RoundRobin = "round_robin" LeastPending = "least_pending" Weighted = "weighted" )
Load-balancing strategies, as accepted in the "load_balancing" config key.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Backend ¶
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
func NewPool ¶
NewPool builds a pool from specs. strategy is one of RoundRobin (the default when empty or unrecognised), LeastPending, or Weighted.
func (*Pool) MarkUnhealthy ¶
func (*Pool) Models ¶
Models returns every model name any backend in the pool serves, sorted and de-duplicated. Backs the OpenAI-compatible GET /v1/models.
func (*Pool) Next ¶
Next returns a healthy backend serving model, chosen by the pool's load-balancing strategy, plus a release func the caller must call once it has finished proxying to that backend (it decrements the in-flight count the least_pending strategy reads; a no-op for the others). It returns an error if no backend serves the model or all of them are currently unhealthy.
Source Files